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Series Foreword 


The John D. and Catherine T. MacArthur Foundation Reports on 
Digital Media and Learning, published by the MIT Press in col- 
laboration with the Monterey Institute for Technology and Edu- 
cation (MITE), present findings from current research on how 
young people learn, play, socialize, and participate in civic life. 
The reports result from research projects funded by the MacAr- 
thur Foundation as part of its $50 million initiative in digital 
media and learning. They are published openly online (as well as 
in print) in order to support broad dissemination and to stimu- 
late further research in the field. 



Introduction 


In 2010, the authors of this report were asked to review the rel- 
evant literature and convene a series of expert meetings to make 
recommendations on the state of the art of, and the outstanding 
challenges in, documenting and assessing learning in informal 
and media-rich environments. 

For several years now, efforts such as the MacArthur Foun- 
dation's Digital Media and Learning (DML) initiative have sup- 
ported the development of a range of educational activities, 
media, and environments outside the classroom and its for- 
mal curriculum. The DML Connected Learning Research Net- 
work has elaborated the principles underlying the evolution of 
an openly networked learning ecology and is conducting stud- 
ies to further define opportunities that support learning across 
contexts (Ito et al. 2013). Other large-scale efforts, such as the 
National Science Foundation-supported LIFE Center (Learning 
in Formal and Informal Environments), have also emphasized 
the complementarity of school and nonschool learning experi- 
ences and the potential for educational reform to benefit from 
knowledge gained in the study of learning outside school. 

In a similar vein, the National Research Council produced 
a consensus report reviewing the knowledge base of science 
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learning in informal environments (Bell et al. 2009), and the 
Noyce Foundation commissioned a report describing the attri- 
butes and strategies of cross-sector collaborations supporting 
science, technology, engineering, and math (STEM) learning 
(Traphagen and Trail 2014). 

In all these efforts, there is agreement that the success and 
expansion of out-of-school initiatives depends on our ability to 
effectively document and assess what works in informal learn- 
ing and what doesn't, as well as where, when, why, and how it 
works. 

This report summarizes an extensive review of the literature 
on the assessment of learning in informal settings, with a focus 
on the following types: 

• After-school programs These activities are not directly meant 
to serve school-based academic functions (e.g., playing an edu- 
cational computer game and making innovative use of it for fun, 
with ancillary learning). 

• Community center programs These activities are negotiated 
between learners and providers. They may have specific learn- 
ing objectives as well as changing approaches to the goal (e.g., 
telementoring and the use of computer simulation of electric 
circuits, along with an on-site coach familiar with the student 
but not responsible for the content). 

• Museum-based programs Visitors can choose to manipulate 
hands-on materials in the context of questions and explanations 
of phenomena observed or produced (e.g., young visitors con- 
necting a battery to various electric devices to see the results of 
completing a circuit, with a coach, and showing the results to a 
parent; or a group of young visitors extracting insects from a bag 
to feed to a pet as part of a long-term project, and one partici- 
pant overcoming a reluctance to touch the insects). 
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• Online communities and forums Participants ask and answer 
questions on a specific area of competence or expertise and 
evaluate one another's answers or contributions. They may also 
engage in joint activity in a virtual space or mediated by tools 
and social interactions in that space (e.g., "modding" in World 
of Warcraft ; learning to build in Second Life ; "theory crafting" to 
identify technical characteristics of computer games by system- 
atically playing many options within them; or raiding as joint 
play for a goal). 

The research review generated an extensive bibliography, 
from which we selected for description and analysis a subset of 
studies and projects to illustrate both the diversity of approaches 
to the assessment of learning in informal activities and good 
assessment practices. 

"Informal learning" is both a broad category and shorthand 
for a more complex combination of organized activities in face- 
to-face or online settings other than formal schools in which 
particular features are especially salient. Characteristically, par- 
ticipants choose and enjoy an informal learning activity for its 
own sake, often engaging in it intensely of their own accord and 
remaining committed to it of their own accord. The power rela- 
tions in informal learning settings typically allow for the rela- 
tively equitable negotiation of learning goals and means. 

The learning goals pursued by participants are generally 
open-ended, dependent in part on available resources and on 
repurposed ways to use those resources. Overall, because of the 
flexibility involved — and the complexity of relationships, means, 
and ends that emerge over time within the activity — many sig- 
nificant learning outcomes may be unpredictable in advance of 
the learner's participation in the central activities undertaken in 
nonformal environments. 
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These features may, in principle, occur in both classroom- 
based learning and other settings, but in different combinations 
and to different degrees. Each setting, and perhaps each kind of 
learning activity, will tend to have a particular combination and 
degree of each feature. The research literature may name activi- 
ties or settings in which these features are present, dominant, 
constitutive, or highly significant (e.g., interest-based learning, 
free-choice learning, nonformal learning, or learning in passion 
communities). The literature may also make distinctions among 
these based on role relationships or types of institutional goals 
and constraints. 

In addition to reviewing the literature, the authors convened 
three expert meetings involving a total of 25 participants to dis- 
cuss key issues, identify successful approaches and outstanding 
challenges, and review summaries of prior meetings in the series. 
The results of these wide-ranging discussions are summarized 
in this report and were highly influential in formulating our 
recommendations . 

Our aim is twofold: first, to offer to those who design and 
assess informal learning programs a model of good assessment 
practice, a tool kit of methods and approaches, and pointers 
to the relevant literature; and second, to offer program staffs, 
project funders, and other supporters recommendations of good 
practices in project assessment and identifiable needs for devel- 
oping improved assessment techniques. 

The members of our expert panels strongly urged us to deal 
with fundamental questions such as the purposes of assessment 
and the kinds of valued outcomes that should be considered. 
From discussions with the panel members and analysis of the 
research literature, as well as our own experience and judgment, 
we constructed a basic assessment model that encompasses at 


Introduction 


5 


least 10 general types of valued outcomes, to be assessed in terms 
of learning at the project, group, and individual levels. Not all 
levels or outcome types will be equally relevant to every project, 
but we strongly believe that all assessment designs should begin 
by considering a conceptual model that is at least as comprehen- 
sive as what we propose here. 

This is particularly important because the valued outcomes 
of informal learning tend to be less predictable and much more 
diverse than those of formal education. Formal education is 
designed to strongly direct learning into particular channels and 
produce outcomes that are specifiable in advance and uniform 
among students. 

Informal learning experiences, in contrast, build on the 
diverse interests and curiosity of learners and support their self- 
motivated inquiries. The valued outcomes of informal learning 
are often particularly rich in contributions to social and emo- 
tional development, to identity and motivation, to developing 
skills of collaboration and mutual support, and to persistence 
in the face of obstacles and in inquiry on time scales of weeks, 
months, and even years. Informal learning activities also often 
result in products and accomplishments of which students are 
justly proud and for which product-appropriate measures of 
quality are needed. 

In the remainder of this introduction, we will present our 
outcomes-by-levels model for comprehensive assessment and 
briefly provide some definitions, distinctions, and principles 
as a general framework for what follows. In the main body of 
the report, we will provide a review of selected and representa- 
tive research studies and project reports in order to illustrate a 
wide range of useful techniques for documenting and assessing 
informal learning across varied settings and to identify issues 
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and challenges in the field. Finally, we will provide our overall 
conclusions and recommendations. 

Outcomes and Levels 

It was universally agreed in our expert panels and extensively 
illustrated in the research literature that simple declarative 
knowledge is only one valued outcome of learning and is too 
often overemphasized in assessment designs to the exclusion or 
marginalization of other equally or more important outcomes. 

Likewise, assessment designs too often focus only on out- 
comes for individual learners and neglect group-level learning 
and project-level or organization-level learning. Documenta- 
tion and assessment must be able to show how whole projects 
and supporting organizations learned to do better or didn't. The 
kinds of documentation and data of value for organizational- 
level improvement are not limited to those that document indi- 
vidual learning. 

Even individual learning is not simply a matter of domain- 
specific knowledge. As an aspect of human development — at the 
individual, group, or organizational level — the learning that mat- 
ters is learning that is used. This type of learning plays a role in 
constructive activities: from posing questions to solving prob- 
lems, from organizing a group to building a simulation model, or 
from exploring a riverbank to producing a video documentary. In 
all these cases, what matters is know-how; "know-that" matters 
only insofar as it is mobilized in practice. Such learning is con- 
sequential and underlies movement, organization, and change. 

Activities of practical value usually require interaction and 
collaboration with other people. "Know-who" is as important 
as know-how in getting things done. Social networking and 
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coming to understand who is good at what, and how a group 
of particular people can work together effectively, is an essential 
outcome of learning. 

Nothing of value can be undertaken unless people are 
motivated to act and feel comfortable within the domains of 
know-how and know-who. A key outcome of learning is the 
development of identification with ideals, goals, groups, tools, 
media, genres, and styles that constitute our changing identities 
and motivations for action. Equally important is our social-emo- 
tional development in learning how to use our feelings — our 
emotional relations to others and our emotional reactions to 
events — for constructive purposes. 

Collaborative groups learn, develop, and change over time. 
Membership may change; agreed-upon goals, processes of inter- 
action, interpersonal feelings, agreed-upon procedures, and 
informal ways of doing things all change. In many cases they 
change adaptively so that the goals of the group are more effec- 
tively pursued. Just as individuals learn how to better function 
in collaborative groups, so groups learn how to make better use 
of the contributions of individual members — or they don't. 

Whole projects, online communities, and larger organizations 
also learn, change, and adapt — or they don't. Documenting and 
assessing organizational learning is equally as important as assess- 
ing group and individual learning and development. It is likely, 
though not well understood, that learning processes at these three 
levels (individual, group, and project or organizational learning) 
are linked and that we cannot expect to understand why learning 
was successful or unsuccessful at any one of these levels unless we 
also have data about learning at the other two. 

From these and similar considerations, we developed the fol- 
lowing basic outcomes-by-levels model for documentation and 
assessment (see table 1). 
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Table 1 

Outcomes-by-Levels Model for Documentation and Assessment 


Level of Analysis 


Project or 



Outcomes 

Community 

Group 

Individual 

Social-emo- 

Developing 

Mutual sup- 

Comfort 

tional-identity 

social-emotional 

port, challenge, 

and sense 

development 

climate; com- 

inspiration; joint 

of agency 


munity or project 

enjoyment and 

in domain; 


ethos, goals, and 

engagement. 

engagement; 


local culture; sys- 


long-term 


tem of roles and 


interest and 


niches. 


persistence 

versus 

obstacles and 
frustration. 

Cognitive- 

Developing 

Shared, distrib- 

Knowing 

academic 

strategies for 

uted know-how; 

how to go 

(know-how) 

organizing and 

collective intel- 

forward in 


distributing 

ligence; dialogue 

the domain; 


know-how; work 

and cooperation 

knowing 


practices and divi- 

skills; explanation how to 


sion of labor. 

skills. 

mobilize and 

integrate 

know-how 

across 

domains. 


In addition to providing this basic outcomes-by-levels matrix, 
we also need to emphasize the importance of taking into account 
in assessment design the incorporation of relevant knowledge 
about the history of the project, the community, and the partici- 
pating organizations and knowledge of the current wider insti- 
tutional contexts (e.g., goals, organization, leadership, resources, 
and limitations). 

We further identified a more specific set of outcomes as rel- 
evant within this overall model, which we have organized into 
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four clusters emphasizing different aspects of learning. First is 
the personal increase of comfort with, and capacity to partici- 
pate in, activities that involve inquiry, investigation, and repre- 
sentation of phenomena in a widening range of domains. This 
set of outcomes emphasizes progressive attunement to the types 
of discourse and practices commonly associated with knowledge 
within a given domain, leading to an increased sense of agency 
and the ability to further leverage resources for learning. 

Second is the improved ability to act collaboratively, coor- 
dinating and completing tasks with others, assisting them, and 
productively using affective sensibilities in doing so. Third is 
learning to critically reflect on the nature and quality of prod- 
ucts and other goal-oriented objectives, becoming able to more 
successfully iterate toward high-quality outcomes. And fourth 
is mobilizing social resources, networks, and capital, including 
across tasks and settings, to reach goals that may take extended 
periods to achieve. 

For each of these four clusters, we include examples of out- 
comes at the project, group, and individual level (see table 2). 

The research projects summarized in the review of the litera- 
ture were selected for inclusion because they provide examples 
of methods for documenting and assessing one or more of the 
above outcome clusters at one or more of the three levels of anal- 
ysis. In the review, we specify at the beginning of each project 
summary the outcomes and levels assessed in each project. 

A Framework of Basic Concepts 

The discussions in our expert panels frequently focused on 
an emerging reconceptualization of key concepts pertinent to 
documentation and assessment design for informal learning 
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Table 2 

Clusters of Informal Learning Outcomes by Level, with Examples 


Outcomes Project Group Individual 


Increasing 
comfort with, 
and the ability 
to conduct, 
independent 
inquiry across a 
widening range 
of domains, 
including 
evaluating 
sources and 
contributions. 

Improving the 
ability to learn 
and act collabor- 
atively, includ- 
ing a relevant 
understanding 
of and support 
for learning 
partners. 

Improving the 
quality of prod- 
ucts, including 
the ability to 
critically reflect 
on the quality 
of one's own 
and others' 
productions. 

Increasing the 
range of social 
resources and 
networking to 
achieve goals. 


The Colorado 
Hybrid Proj- 
ect's cultural 
responsiveness 
promotes girls' 
identification 
with STEM 
practices. 


The GIVE 
project prompts 
groups of inter- 
generational 
museum visi- 
tors to engage 
in inquiry. 


Digital Zoo, a 
game designed 
for the devel- 
opment of 
students' engi- 
neering epis- 
temic frames. 


Programming at 
the 5th Dimen- 
sion sites is sus- 
tained through 
partnerships 
and the scaling 
of practices. 


Families' 
scientific sense 
making at a 
marine park, 
demonstrated 
by TOBTOT. 


Children 
engage in 
mutual helping 
behavior at 
5th Dimension 
sites. 


Youth shape 
one another's 
programs in 
Computer 
Clubhouses. 


YouMedia par- 
ticipants work 
with peers to 
create products 
relevant to their 
shared interests. 


Use of Zydeco 
for "nomadic 
inquiry" of con- 
tent across learn- 
ing settings. 


Collaborative 
problem solving 
for success in Lin- 
eage and other 
MMORPG play. 


DUSTY par- 
ticipants develop 
agentive capacity 
in creating their 
digital life stories. 


WINS par- 
ticipants in 
a museum 
program draw on 
resources to help 
support STEM 
career paths. 
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activities. There was broad consensus across the three expert 
meetings on how to employ the terms elaborated below, but the 
report's authors assume responsibility for the specific formula- 
tions provided here. Some key terms in the individual project 
studies reviewed in this report will be used differently from how 
we use them. We will try to make this difference clear in each 
case while otherwise maintaining our own consistent usage of 
the following terms: 

Learning Learning that matters is learning that lasts and that 
is mobilized across tasks and domains. Our notion of learning 
includes social-emotional-identity development as well as know- 
how and know-who; it should also include learning by groups 
and communities or organizations as well as by individuals. 
Knowledge Knowledge that matters is knowing how to take 
the next step, for which declarative knowledge is merely one 
subsidiary component and greatly overemphasized in current 
assessment. Know-that matters only insofar as it is mobilized 
as part of know-how; know-how (cultural capital) matters for 
career futures and social policy only when effectively combined 
with know-who (social capital). The social networking aspects of 
relevant knowledge are underemphasized in current assessment. 

Know-how and other aspects of knowledge have to be 
defined for groups and communities as well as for individuals. 
Groups and communities always know more, collectively, than 
any individual member knows, and collective intelligence and 
problem-solving skills, creativity, and innovation are also gener- 
ally superior to what individuals are capable of. 

Assessment The production of knowledge useful for individu- 
als, groups, and communities to improve practices toward val- 
ued goals; distinguished from evaluation. 
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Evaluation Judgments made on how well goals are being 
achieved and how valuable the totality of all the outcomes is. 
Research The production of knowledge useful for the design 
of activities and communities capable of reaching stated goals 
and with enhanced potential for producing valuable outcomes 
beyond stated goals. 

Documentation The collection of information useful for assess- 
ment, evaluation, and/or research. 

Assessment, evaluation and research all build on documen- 
tation but may require different modes and foci of documen- 
tation. In more traditional terms, assessment aims at locating 
outcomes, evaluation aims at judgments about effectiveness and 
directions for improvement, and research aims at generalizable 
knowledge that may be used for future design. 

Engagement Affective involvement in and commitment to an 
activity, goal, practice, group, or community that enhances the 
quality and quantity of participation despite obstacles, setbacks, 
or frustrations; distinguished from enjoyment. 

Enjoyment The positive feeling accompanying an activity that 
makes it worth doing for its own sake. Both engagement and 
enjoyment are important aspects of learning and should be doc- 
umented in assessment, although it should be recognized that 
negative feelings may also play a significant role in engagement 
and in learning. 

Agency This term has several different meanings: actual effec- 
tiveness; a disposition toward taking action; a feeling of self- 
efficacy; and an aspect of one's identity as someone who can 
produced desired effects. All these meanings are task- and/ 
or role- and domain-specific, and also often group- or com- 
munity-specific. The notion of agency also extends to what a 
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group or community believes it can accomplish or can actually 
accomplish. 

Outcome Conventionally, an (occasionally naive) attribu- 
tion of a valued condition to some specific cause (e.g., to an 
intervention). Rarely, however, are valued learning goals the 
outcome of discrete, identifiable causes. Moreover, posited out- 
comes observed at some moment in time or over a short interval 
do not necessarily persist or serve as a foundation for further 
development. They are frequently transitory phenomena, arti- 
ficially produced by the procedure used to measure them. We 
will instead use the term outcome to refer to socially and person- 
ally valued ongoing processes that emerge in the milieu of some 
community and its activities. Note that we regard evidence of 
learning in progress as equally important as evidence of com- 
pleted or stabilized learning. 

Unit of analysis What should be the unit in focus in assessment 
design? We believe that it should be a system of activities and 
practices over time; these include the actions of individual learn- 
ers as well as the roles of other participants, such as mediating 
tools, semiotic media, and local conditions directly relevant to 
and supportive of (or obstructing) the learning activities. The 
unit of analysis must be extended peripherally to wider contexts 
that make the learning activity possible institutionally, but with 
decreasing detail as their relevance to the specifics of learning 
trajectories decreases. Assessment at the level of individuals, 
groups, and whole projects are necessarily interdependent, and 
assessment design must include all three and their relations to 
one another. 



Review of the Literature 


This section provides a detailed review of selected studies and 
projects in several domains of informal learning. 

What is meant in this report by informal learning builds on 
a fundamental assumption: Learning in its broadest sense takes 
place in every activity of life insofar as what we do at a later time 
benefits from earlier experience. Informal learning is, more spe- 
cifically, learning as an outcome of participation in organized 
activity that is characterized by its more voluntary, interest- and 
enjoyment-based, and open-ended nature compared to formal 
instruction. 

This definition still leaves the potential scope of informal 
learning too broad for the purposes of this report. Within such 
a scope, we would have to include domains such as organized 
sports activities, learning in practical activities in the home, and 
leisure activities developed by organizations such as the Boy 
Scouts — all of which we exclude. The domains that are included 
in this research review are the following: 

• Learning in after-school programs and community centers 

• Short-term, focused, out-of-school activities 
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• Activities in informal learning institutions (e.g., museums, 
aquariums, and zoos) 

• Computer-based and online activities 

For each domain or type of setting, this review looks at exem- 
plary studies and projects, identifying particular approaches to 
assessment, methods of documentation, and techniques of anal- 
ysis. The types of learning outcomes investigated by the original 
authors and the levels (individual, group, and organizational) 
referenced by them are identified. After the review, we include 
summaries of the discussions in the three expert meetings. Fur- 
ther details on the expert meetings and other resources identi- 
fied during the project are available on the project wiki at http:// 
documentinglearningworkshop.wikispaces.com. 

After-School and Community Centers 

In this section, we discuss examples of after-school programs 
run in a variety of locations and with a variety of educational 
objectives. 

A common source of social and scientific interest in edu- 
cational programs conducted during nonschool hours arises 
specifically because the learning activities are not occurring in 
school. Although they vary greatly, the community-based pro- 
grams described in this section all seek to leverage the fact that 
after-school settings are more flexible in schedule and social 
arrangement, allowing children the freedom to engage in peer 
interaction and less hierarchically codified interactions with 
adults. Because the activities take place between the home and 
the school, at a time of day that — since the advent of modern 
schooling — has given play a privileged position, the settings are 
places where having fun is an essential ingredient and where, to 
some extent, children participate voluntarily. 
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This shared circumstance offers both the greatest promise 
and the greatest challenges to realizing the hope placed on after- 
school activities for infusing meaningful learning into the lives 
of children. On the positive side, the organization of after-school 
settings is centered on activities that children and youth will 
want to engage in voluntarily; if you build it and it's unattract- 
ive, the kids might not come. One particular draw is that after- 
school programs generally allow children and youth to work in 
groups with their peers and to choose the roles that they play in 
the various projects that are offered. Indeed, children and youth 
often have a voice in the projects that are offered or are allowed 
to walk away from those they find boring. 

In this kind of social environment, children are free to speak 
with one another, often using the language they feel most com- 
fortable speaking, and the staff members (who vary in age from 
their late teens to middle age) are freed of the obligation to know 
all the right answers; this allows them to position themselves in 
the roles of coaches and more experienced peers. As a result, par- 
ticipation is also a learning experience for the adult staff, includ- 
ing older youth. 

Unfortunately, many of the characteristics that offer the 
greatest potential for after-school activities to promote learning 
are the same characteristics that offer the greatest challenges for 
using after-school settings to provide the kind of educational 
experiences for which they appear to be perfectly suited. For 
instance, because after-school activities are not part of the for- 
mal school system and children are not legally bound to attend, 
the funding sources to support such settings are scarce and 
uncertain; so too is the consistency of attendance by the school- 
age population such settings are designed to serve. For the same 
reason, few staff members earn a living wage from this work, 
which leads to a high rate of staff turnover and a low level of 
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education among the staff members with whom children and 
youth are in close contact. 

These "structural" factors (Bevan and Michalchik 2012) go 
hand in glove with the promising social-organizational charac- 
teristics of after-school activities summarized above. If a learner's 
project is unfinished, it might not be taken up again in a later 
session, which creates difficulties for programs with goals, such 
as promoting science engagement. Projects often do remain 
unfinished, sometimes even unstarted, as children, restless after 
a long day of sitting in enforced quiet in school, are freed up 
to have fun in ways that have nothing to do with preplanned 
activities. 

For our present purposes, a major challenge of informal, vol- 
untary, peer-oriented, group-organized, after-school programs 
is that they are difficult to evaluate. Although evidence-based 
assessment has merely meant randomized trial research designs, 
as federal regulations have often been interpreted in evaluat- 
ing formal education, the voluntary nature of informal learning 
activities precludes random assignment and makes the persis- 
tence of learners in a trial unreliable. Participants come and go, 
appear for some sessions but not all, change activity groups as 
their friendship networks evolve, and leave the activity if their 
participation is too controlled. Standardized assessments, more- 
over, are based on the assumption that the valued outcomes of 
an activity can be known in advance, whereas experience with 
informal learning programs shows that some of their most val- 
ued outcomes emerge unexpectedly. 

Each of the programs reviewed here provides useful sugges- 
tions for ways to assess informal learning activities. A key lesson 
they teach is that one size does not fit all. As in the programs 
themselves, a variety of assessment strategies are the rule, not 
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the exception (e.g., Bell et al. 2009; Bransford, Brown, and Cock- 
ing 2000; Harvard Family Research Project 2008; Mahoney, Lar- 
son, and Eccles, 2005; Michalchik and Gallagher 2010; Shaffer 
and Gee 2012). 

We have chosen to focus on several programs that have 
achieved sufficient scale to provide a foundation for reaching 
conclusions that are potentially generalizable across a reason- 
able range of variation. These programs share the following 
characteristics: 

• A focus on leveraging new technologies for learning after 
school 

• A university-community partnership model for after-school 
learning and research 

• The use of field notes, interviews, video analysis, and other 
site-specific evidence of learning and development 

The 5th Dimension: A Broad-Based Enrichment Program 

Levels of Analysis Individual, group, project 

Valued Outcomes Improved literacy and numeracy, agentive 

participation, long-term sustainability 

Methods Cognitive-ethnographic field notes of adult (includ- 
ing college student) participants, videography, quantitative data 
collected as part of normal practices 

The 5th Dimension is an educational activity system that 
offers school-age children a specially designed environment in 
which to explore a variety of off-the-shelf computer games and 
gamelike educational activities during after-school hours. (The 
account here draws heavily on Cole and the Distributed Literacy 
Consortium 2006.) 
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The 5th Dimension involves the participation of undergradu- 
ates enrolled in a practicum course and elementary school-age 
children in a community setting. Such settings are organized so 
that adults provide for the involvement of children in presum- 
ably prosocial, development-enhancing activities in nonschool 
hours in spaces that can house such activities (e.g., Boys and 
Girls Clubs, YMCAs and YWCAs, local libraries, and after-school 
programs in a school, church, or community center). 

The 5th Dimension was modeled loosely on the idea of the 
game Dungeons and Dragons. The conceptual layout of the 5th 
Dimension includes a labyrinth that contains a variety of com- 
puter and noncomputer games that are part of a make-believe 
world. The 5th Dimension's materials and conventions are orga- 
nized to achieve the kinds of objectives enumerated above. By 
intertwining fun and academic motivations, the activities pro- 
mote children’s engagement in reading and writing. 

At the same time, this play world is designed to help partici- 
pants (both children and undergraduate students) orient to the 
game's attractions and challenges, to form goals, and to chart 
progress toward becoming an expert. Other features are designed 
to provide motivation to write to someone, to look up informa- 
tion in an encyclopedia, to teach someone else what one has 
learned, and to reflect upon and criticize information. 

As a means of balancing play and work, 5th Dimension sites 
often have an electronic entity (a wizard or wizardess) who lives 
on the Internet and writes to (and sometimes chats with) the 
children and undergraduates online. In the mythology of the 
5th Dimension, the wizard(ess) acts as the participants' patron, 
provider of games, mediator of disputes, and source of computer 
glitches and other misfortunes. 
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Because it is located in a community institution, 5th Dimen- 
sion requires a local site coordinator who greets the participants 
as they arrive and supervises the flow of activity in the room. The 
site coordinator is trained to recognize and support the pedagog- 
ical ideals and curricular practices that mark the 5th Dimension 
as different — that is, a different way for kids to use computers, a 
different way of playing with other children, and a different way 
for adults to interact with children. 

The presence of university students is a major draw for the 
children. The participating college students are enrolled in a 
course focused on fieldwork in a community setting where they 
act as buddies for the children. It is in these cross-generational 
collaborative activities, organized around solving puzzles and 
playing games, that the socioemotional aspects of the children's 
behavior are most clearly visible. 

As a result of the wide range of institutional settings in which 
5th Dimension has been implemented, an equally wide range 
of assessment strategies has been used (see Cole and the Distrib- 
uted Learning Consortium 2006). Some of these strategies have 
involved a focus on outcomes of learning, whereas others focus 
on evidence of learning in process. Some focus on individual chil- 
dren, some on the success of the activity within its community 
setting (judged by children's attendance and levels of enthusi- 
asm), and some on the degree of support and participation by the 
cooperating institutions. The particular mixture of methods cru- 
cially depends, as does the activity itself, on its context. 

The outcomes-based assessments focused on individuals are 
most likely to be useful in situations where the local social ecol- 
ogy permits random assignment of children to participation in 
the 5th Dimension. In such cases external criteria (i.e., statewide 
achievement tests provided by the schools as well as various 
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tailored pre- or post-test tasks given as a condition for partici- 
pation) have been implemented. In situations where no proper 
control group can be included but where there are more children 
who desire to participate than the facility can handle, various 
quasi-experimental tasks using 5th Dimension activities have 
been implemented. 

Evaluations at this individual-focused level have included the 
following (Cole and the Distributed Learning Consortium 2006): 

1. Computer literacy Paper-and-pencil computer knowledge 
tests, evaluation of memory for computer terminology, and 
hands-on computer-use proficiency merit badges. 

2. Mathematical understanding and mathematical problem solv- 
ing Understanding arithmetic word problems isomorphic to 
those in the computer games, and using math problem-solving 
strategies in the games themselves as evidenced by field notes. 

3. Reading and linguistic skills Reading comprehension of 
instructions in novel games, notes read from and written to the 
wizard, live written chats with the Wizard, and success in a vari- 
ety of record-keeping practices. 

Since it is impractical to create control or comparison groups 
for group-level outcomes, researchers have used in situ, process- 
focused evidence of children taking greater control of the con- 
duct of the activities; longitudinal studies of children's progress 
from beginner to excellent levels of performance within games; 
and data mining of large samples of field notes to reveal the fre- 
quency and sophistication of academic skills, such as reading, 
and of social skills, such as helping others. 

The criteria for assessment of success at the level of the activi- 
ties within cooperating institutions include whether supervisory 
personnel who are supported by the community organization 


Review of the Literature 


23 


participate in the activities at their site, and whether the uni- 
versity supports continuing practicum classes that supply super- 
vised undergraduates as more capable peers or as buddies to 
work and play with the children and write detailed field notes. 
To attain the long-term goal of a sustainable new cultural prac- 
tice, the cooperation of the implementing institutions in obtain- 
ing the additional funding required for the transaction costs of 
the collaboration provides a strict criterion for this objective. 

For dozens of 5th Dimension programs, mixtures of appropri- 
ate evaluations of the kind sketched above have shown that the 
programs are effective at the individual level. Children like the 
activities and participate; they are generally agentive, argumen- 
tative, and engaged in observable and documentable ways. In 
many, but not all, of the social contexts in which the innovation 
has been tried, adults approve of the activities and make their 
children available. College students show marked improvement 
in understanding how to be effective supporters of children's 
and their own learning. At the group level, there is evidence of 
improved group collaboration and task success (Cole and Distrib- 
uted Literacy Consortium 2006; Downing- Wilson 2006/2007). 

One of the instructive features of the 5th Dimension project 
is the extent to which it has afforded assessment of learning at 
the level of the program as a whole in its socioecological con- 
texts. No 5th Dimension program is a static, unchanging system 
of activity; instead, each changes in relation to itself and its insti- 
tutional settings, which are themselves changing. For example, 
at one implementation of the 5th Dimension at a Boys and Girls 
Club over a 16-year period (Downing-Wilson et al. 2012), the 
careful documentation process reveals a dynamically changing 
relation between the 5th Dimension itself and its host institu- 
tion. Initially, the 5th Dimension is more or less self-contained 
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in an accessible room. Then, over time, in response to changes 
in club policies and resources, the practices of the 5th Dimen- 
sion gradually start to infuse all areas of club activity, engaging 
a wider range of children, involving club personnel more deeply 
in the program, and enlarging the role of the 5th Dimension in 
its setting. 

Other 5th Dimension programs show different patterns of 
learning at the project and organizational level as they adapt 
to rapid changes in computer technology, turnover in person- 
nel, changing funding streams from a variety of sources, and 
changing priorities of the collaborating institutions. It is no sur- 
prise that there are failures as well as successes in adapting. Many 
5th Dimension projects run successfully for two, three, or more 
years, only to fall apart because of changes in institutional prior- 
ities or the departure of key personnel. Others manage to adapt 
to the changed conditions and have continued for many more 
years. At the time of this writing, some such systems have been 
in existence for more than 15 years, permitting analysis of the 
institutional and social-ecological conditions for sustainability. 

The Computer Clubhouse Network: Learning to Program 

Levels of Analysis Individual, group, project 

Valued Outcomes Collaboration, quality of products, social 

resources 

Methods Case study, Oakes’s model of school reform, portfo- 
lios of activities 

The Computer Clubhouse Network is another after-school 
program similar in organization and goals to the 5th Dimen- 
sion program. Started in 1993 through a collaboration between 
MIT's Media Lab and Boston's Computer Museum, the network 
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currently consists of more than 100 after-school learning cen- 
ters dedicated to providing primarily high-end digital media 
instruction to help elementary school-age children through high 
school-age youth develop the expertise to express themselves flu- 
ently through new technologies (Resnick, Rusk, and Cooke 1998). 
To create and sustain an environment in which this communica- 
tive fluency is developed, Computer Clubhouses are designed to 
cultivate an "emergent community" that promotes opportunities 
for youth to engage in design projects that are driven by their 
own interests (Kafai, Peppier, and Chapman 2009). 

Of the several innovative and technology-centered learning 
activities that have been developed and field-tested in Computer 
Clubhouses, the Scratch programming activity has generated 
particular attention. 1 Scratch is a programming environment 
designed to help novices learn programming through easy desk- 
top manipulation of digital images and sound files. Rather than 
type code, learners program by dragging and dropping blocks of 
preset programming commands in a jigsaw puzzle-like fashion, 
creating stacks of blocks. The commands control the actions of 
movable spritelike figures, each of which contains its own set of 
images, sounds, variables, and scripts (see figure 1 for an image 
of the Scratch interface). 

Kafai, Peppier, and Chiu (2007) examined the organization 
of activities at a Computer Clubhouse before and after the intro- 
duction of Scratch as one of the activities in order to understand 
how after-school learning environments might be seeded with a 
computer-programming activity. Kafai and her colleagues drew 
on the model of school reform offered by Oakes (1992). Accord- 
ing to Oakes, efforts to reform educational activities in a man- 
ner that ensures they are equitable must go beyond the technical 
aspects of implementing these activities. It is also necessary to 
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Figure 1 

Screen captures of the Scratch program interface. Images courtesy of 
Mitchel Resnick. 
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encourage changes in the normative and political dimensions of 
each activity — that is, its longstanding norms, values, and insti- 
tutional support within the larger community. 

Kafai et al. (2007) documented and analyzed the organization 
of activities at two Computer Clubhouse sites in Southern Cali- 
fornia. At the technical level, the researchers were interested in 
observing how, and to what degree, Scratch would be adopted 
as an activity, given specific conditions of implementation. 
These conditions included the incorporation of new activities 
like workshops and gallery presentations and the participation 
of undergraduates who acted as mentors but who knew little 
to nothing about programming. Parallel to the principles of the 
5th Dimension model, the inclusion of inexperienced program- 
mers as mentors was seen as contributing to the occasioning of 
opportunities for mentees to teach the novice mentors about 
programming in Scratch. 

Normatively, the focus was on how participants (including 
youth, mentors, and parents) interpreted the meaning and value 
of computer programming (e.g., what participants considered 
prototypical programming projects) and how they interpreted 
the value of their own programming abilities. In the political 
dimension of the analysis, Kafai et al. (2007) reported the way 
in which a formal partnership developed between the univer- 
sity and the Computer Clubhouse's host organization. These 
changes were crucial, the researchers noted, in gaining the nec- 
essary support to create the infrastructure for the Computer 
Clubhouse's goal of achieving technological fluency. 

The primary documentation data at the individual and group 
levels were researcher and mentor-produced field notes and 
sample products of participants' efforts. To address technical 
and normative questions about how Scratch was incorporated 
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in the clubhouse and the role of the mentors in this process, the 
field notes were coded for different kinds of activities: design, 
Web, and personal (e.g., socializing). The resulting products 
at the clubhouse, consisting of projects of all kinds (including 
Scratch programs), were analyzed based on project type, such as 
animation, game, story, and graphics (Kafai et al. 2006). In addi- 
tion, group interviews of clubhouse members and undergradu- 
ate mentors were conducted in order to capture how the young 
participants understood the development of their own program- 
ming skills. 

From the analysis of field notes, projects, and interviews, 
portfolios of activities were generated to show changes in the 
kind and number of activities the youth engaged in before and 
after the introduction of Scratch. The identification of periods 
of increased production guided more focused analyses of the 
kind and distribution of projects. These steps, in turn, allowed 
Kafai et al. (2006) to identify and conduct case studies of Com- 
puter Clubhouse members who created projects, the structure 
and content of which appeared to influence the production of 
Scratch projects by other clubhouse members. (When we con- 
sider computer programming as an activity in itself in a later 
section of this review, we will return to examine how researchers 
also tracked the quantity and distribution of programming com- 
mands in Scratch projects throughout a particular period). 

Digital Storytelling and Media Production Programs 

In this section we examine two projects that seek to create envi- 
ronments in which youth gain exposure to new, academically 
relevant activities and technologies in ways that encourage the 
incorporation of the learners' interests. Both examples focus on 
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assessing learning through careful examination of the process 
and products of youth efforts to produce digital media. 

The first case, a digital storytelling after-school program, illus- 
trates ways in which extensive ethnographic data, including par- 
ticipant-produced media, can be analyzed for insights into the 
development and enactment of the learner's authorial agency. 
The second case is an example of how data visualizations can 
be used to assess learning in participants' uses of online social 
media platforms in a technology-driven after-school program. 

The Digital Underground Story Telling Youth Project 
Levels of Analysis Individual, learning ecologies 
Valued Outcomes Independent inquiry, collaboration, social 
resources, identity formation 

Methods Case study, thematic analysis, narrative analysis, per- 
formative moments 

The Digital Underground Story Telling Youth (DUSTY) proj- 
ect is an adult and youth multimedia literacy program run out 
of a community technology center located in the Bay Area of 
California. DUSTY differs from traditional, academically ori- 
ented after-school programs in its emphasis on the centrality of 
identity formation and meaning making in learning, particu- 
larly the role that semiotic systems in addition to language play 
as resources for "embodying and enacting a sense of self in rela- 
tion to others" (Hull and James 2007). To this end, the creation 
of digital stories at DUSTY is aimed at positioning participants to 
use narrative reconstruction to reflect on their experiences and 
to be active agents in articulating their own aspirations. 

This agency- and identity-centered framework for theorizing 
and assessing learning is described in detail by Hull and Katz 
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(2006). The authors emphasize the importance of the various 
social contexts of learning (e.g., the DUSTY program, the school) 
and their contributions to a learner's digital story production. 
They argue that people can use their personal repertoire of tools 
and other resources to develop agency by creating "multimedia 
autobiographical narratives about self, family, community, and 
society ... to articulate pivotal moments in their lives and to 
assume agentive stances toward their present identities, circum- 
stances and future" (44). 

Hull and Katz (2006) report longitudinal case studies of youth 
participants using the following as documentation: field notes 
of participant activities in multiple settings, written by research- 
ers and undergraduate mentors; interview data to characterize 
participant histories inside and outside DUSTY; and participant- 
produced media such as story boards, scripts, and digital stories. 

To demonstrate how two such participants developed autho- 
rial agency through their creation of digital stories, Hull and 
Katz (2006) drew on a variety of concepts, originally proposed 
by scholars such as Bruner (1994), Urciuoli (1995), and Bauman 
and Briggs (1990), to identify displays of agency both in these 
youths' actions (captured through field notes and interview tran- 
scripts) and in their digital stories. They employed Bruner's idea 
of narrative turning points, when people "report sharp change 
in their lives and demonstrate accompanying dramatic changes 
in their representations of self" (Hull and Katz 2006, 45). From 
Urciuoli they adopted the idea of performative moments, which 
include "any activity that coordinates action to create a unity 
from many selves” (47). From Bauman and Briggs they adopted 
an agent-centered view of verbal performance, in which agency 
is exercised by a person through the decontextualization and 
recontextualization of verbal texts, and applied it to the multiple 
modalities of digital stories (see figure 2). 
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Figure 2 

Excerpt of a graphic representation of a digital story by a participant 
named Randy. The representation was used as a tool to identify and 
analyze patterns relevant to the focus of analysis in Randy's case, 
authorial agency. The representation juxtaposes screen shots from the 
digital story, text of the corresponding voice-over, and excerpts of rele- 
vant comments from interviews. From Hull and Katz (2006), page 51. © 
2014 by the National Council of Teachers of English. Reprinted with 
permission. 

The case study subjects not only mastered the technical skills 
required to create digital stories, they also became more sensi- 
tive to the genres and poetic aspects of language, and they grew 
increasingly adept at combining multiple media (text, sound, 
and images) to create personally relevant narratives. 

YouMedia and iRemix: Data Visualizations in Online Networks 
Levels of Analysis Individual, group 

Valued Outcomes Independent inquiry, collaboration, quality 
of products 
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Methods Social learning network analysis, timeline structures 
for data visualizations 

YouMedia is another example of a program dedicated to 
providing youth with the space, resources, and semistructured 
opportunities to use high-end digital media to explore new 
modes of self-expression. Housed in a centrally located urban 
library, YouMedia is a collaborative project of the Chicago Public 
Library and the Digital Youth Network, a Chicago-based digital 
literacy project that provides linked in-school and after-school 
media classes (http://www.digitalyouthnetwork.org). At YouMe- 
dia, teens from all parts of Chicago have access to a rich array 
of technological and social resources, including laptops, smart 
boards, video cameras, and an audio recording studio. 

Based on similar theoretical principles as DUSTY (described 
above), the activities at YouMedia are organized to encourage 
teens to identify topics of interest and to explore telling stories 
about these topics through multiple media. Mentors from the 
Digital Youth Network visit YouMedia to share their media pro- 
duction expertise with the teens, who work on digital activities 
in music, design, photography, blogging, and video production. 

Teens who wish to participate in YouMedia activities must 
register as members. As part of this process, the teens are invited 
to join a cloud-based, online social network, YouMedia Online, 
restricted to and accessible only by YouMedia members . 2 Users 
of YouMedia Online can do things that are common on social 
networking sites, such as managing their profile pages, posting 
comments and media, and joining groups. 

However, YouMedia Online differs from other social net- 
working sites by being designed to support both structured 
and self-directed learning in ways that connect to the culture 
and expectations of the YouMedia community (Austin et al. 
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2011). For example, one of the ways that learning is promoted is 
through the use of a virtual currency called Remix dollars. These 
dollars are awarded for, among other things, posting original 
content and providing feedback to others by posting comments 
on their projects. This currency can then be redeemed for such 
items as gift cards, USB drives, and MP3 players. 

As a repository of vast amounts of information about the 
teens' YouMedia-related activities, the iRemix platform offers a 
rich source of data for exploring questions about learning and 
the development of expertise, interests, and social capital. Nich- 
ole Pinkard, who founded the Digital Youth Network, and her 
colleagues have taken the first steps in exploring the mechanics, 
ethics, and potential insights of mining data from social learn- 
ing networks like the iRemix platform (Nacu et al. 2012). They 
have examined ways of adapting and applying to the iRemix 
platform the tools, metrics, and analytic frameworks typically 
used in the private sector for studying the use of online social 
networks. 

The questions that drive Pinkard and her colleagues research 
center on the ways youth participate in the environment and 
interact with others, with particular attention on the way these 
interactions lead to learning. Both peers and mentors play a role 
in online learning interactions. Pinkard and colleagues have also 
tracked the relationship between actions such as media viewing 
or user profiles and subsequent production-oriented actions such 
as posting comments, critiques, media ratings, and original work. 
In addition, they have identified features of social learning net- 
works that can be studied to address questions about the kinds of 
learning that unfold for users of these networks (see table 3). 

In order to make sense of the large amount of documenta- 
tion generated from logs of iRemix activity, Nacu et al. (2012) 
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Table 3 

Features of Online Social Learning Networks and Their Corresponding 
Forms of Data 


Participation 

Consumption 

Contribution 

Production 

Access and 
membership 

Views of specific 
items 

Content posted 
to the network 

Posting of origi- 
nal media 

Impact 

Relationships 

Expertise 

Interests 

Evidence of 
impact by 
individuals 

Social ties, 
interactions, 
and group 
affiliations 

Evidence of skill 
development 

Emergence and 
development of 
interests 


Source: Nacu et. al. (2012) 


developed operational definitions of actions and concepts they 
deemed relevant for addressing their research questions (see table 
4). Using these definitions as tools for coding the data, they exper- 
imented with a number of techniques to visualize the results. 

For example, to explore questions concerning the role that 
peer-peer and peer-mentor relationships play in facilitating 
learning, social network maps were generated to chart these 
relationships and their strengths at specific points in the history 
of the program and over time. Another technique creates time- 
line structures that show the kinds of activities individual par- 
ticipants have engaged in over extended periods and also how 
often and for what amount of time they have been involved. 3 

Figure 3 shows an example of a timeline structure for a Digital 
Youth Network participant's online activities during her junior 
and senior years in high school. The timeline covers a period of 
18 months and is divided into three rows. The top row identi- 
fies the projects and workshops that the student participated in, 
with indications of the beginning and end of her participation. 
The middle row specifies activities (named on the far left) that 
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Table 4 

Definitions of Actions and Concepts Determined Relevant for Research 


Questions 

Social Learning Network Features 

Kinds of Data 

Structures to support informal 
and formal interaction among 
users 

Social ties 

Access to teachers, mentors, and 
experts 

Interaction patterns among peers, 
teachers, and mentors 

Asynchronous, online 
communication 

Access and participation patterns 

Creation, sharing, and discussion 
of multimedia content 

Engagement with 
content;connecting use patterns 
with patterns of learning and 
development 

Ability to structure learning activi- 
ties and projects 

Engagement and impact of specific 
learning resources 


Source: Nacu et al. (2012) 


the student engaged in and that could be extrapolated from the 
students' activity on the iRemix platform (e.g., editing, upload- 
ing and commenting on videos and photos, posting blog entries, 
joining groups), with the size of the circle indicating the number 
of digital products uploaded. The bottom row describes badges 
that could potentially be awarded to the student for engaging in 
activities in a manner deemed successful or valuable by the local 
community. 

STEM-Focused Community-Based Programs 

The projects discussed in this section differ from those above by 
having been designed with specific content-area learning goals. 
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In each case, the content area was in the domain of science, tech- 
nology, engineering, mathematics (STEM), or computer science. 

City Farmers: Emergent Learning in a Community Garden Program 
Level of Analysis Individual 

Valued Outcomes Independent inquiry, social resources 
Methods Case study, semantic domain analysis, discourse 
analysis 

Rahm (2002) conducted an ethnographic study of an eight- 
week summer community youth program designed to teach 
plant science and entrepreneurship and develop the participants' 
teamwork skills. Based at a community garden site in an ethni- 
cally diverse, low-income, inner-city neighborhood, the program 
was structured so that participants rotated through two-week 
cycles in four kinds of activities: nurturing (soil preparation, 
planting, and watering), harvesting (identification, harvesting, 
and preparation of harvestable crops), marketing (contacting 
potential buyers, selling, and organizing delivery), and special 
projects (community outreach, artwork, and tree planting). 

The aim of the study was to examine how opportunities to 
learn science emerged in the everyday interactions among the 
program participants. To this end, Rahm focused her research on 
the language and discursive practices that the participants used 
to make sense of program-related activities. The participants 
included the middle school students enrolled in the program as 
well as the adult members of the program including four team 
leaders, two Master Gardeners, and the program director. 

As a participant-observer, Rahm simultaneously videotaped, 
kept field notes, and worked alongside a team of six 11- to 
14-year-old participants as they engaged in the nurturing and 
harvesting activities. After the data collection phase, transcripts 
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were made of the videos and integrated with the corresponding 
field notes. Rahm subjected these materials to domain analyses 
(Spradley 1980) in order to develop taxonomies of the kinds of 
learning opportunities supported by the program. These analy- 
ses revealed some of the discursive strategies that participants 
used to make meaning of the gardening activities (e.g., analo- 
gies, elaborations, questions, and comments). Rahm found that 
question strategies could be further subdivided into information 
questions, knowledge integration questions, and inquiry. In 95 
percent of the cases studied, inquiry and knowledge integration 
questions were precursors to learning opportunities. 

Rahm also explored the connections that the participants 
made between their experiences in the program and their experi- 
ences with science in school. She conducted pre- and post-exper- 
imental semistructured interviews of six participants whom she 
shadowed and of several other participants selected from the 
teams in the program. The interviews focused on the partici- 
pants' notions of science and on their perspectives on learning 
science in the program and at school. 

Post-experimental interviews were conducted with all of the 
adult program participants in order to document their perspec- 
tives on the learning that takes place in the program. The inter- 
view data revealed that both the youth and adult participants 
saw the activities of the program as providing authentic oppor- 
tunities for learning science that were markedly different from 
those offered in the classroom. 

The Colorado Hybrid School-Community STEM Project 

Levels of Analysis Group, project 

Valued Outcomes Independent inquiry, collaboration 

Methods Design experiment, semantic domain analysis, 

vignette analysis 
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Eisenhart and Edwards (2004) implemented and studied 
an after-school program designed to promote participation in 
computer technology and science activities by urban, African 
American, middle school girls. This university-community col- 
laboration involved university researchers, representatives of 
community institutions, and local residents. The program also 
involved the iterative design and implementation of culturally 
sensitive science and technology-focused classes in the after- 
school hours. Of the cases discussed thus far, this is the only 
one in which a deliberate effort was made to create a learning 
activity that met state curriculum content standards while also 
incorporating the participants' own goals and interests. 

This research was organized to examine two issues: (1) the 
extent to which the participating girls' gender and ethnic identi- 
ties were represented in the technology-driven activities of the 
program, and (2) the implications of this for understanding how 
to successfully engage the girls. The research asked these ques- 
tions: What were particularly good examples of occasions when 
the girls seemed motivated to learn more science or technology? 
What did these occasions suggest about the conditions for suc- 
cessful work with the girls? 

To address these issues, Eisenhart and Edwards (2004) fol- 
lowed a group of six African American girls (12 to 15 years old) 
for one academic year as they participated in science and tech- 
nology activities in the after-school program. Documentation 
of these activities included researcher- and instructor-produced 
field notes and journals, audio and video transcripts of each 
session, audio transcripts of student interviews, the digital and 
concrete products (e.g., T-shirt designs) created by the students 
using the software provided to them, and computer logs of the 
students' activities with these software applications. 
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To address the degree to which these girls' identities were 
reflected in the science and technology activities, Eisenhart 
and Edwards (2004) drew on Spradley's (1979, 1980) semantic 
domain analysis. The basic procedure involves identifying and 
coding semantic domains (participants' units of meaning) and 
constituents (the items that constitute the domains) through the 
application of nine universal semantic relationships to the data 
texts. Eisenhart and Edwards examined three domains that reveal 
how the girls marked gender identity, ethnic identity, and tech- 
nological expertise: (1) statements and actions regarding interest 
in computers and technology, (2) actions regarding female iden- 
tity, and (3) statements and actions regarding African American 
identity. By reviewing the contents of each domain and then 
examining the overlap between domains 1 and 2 and domains 1 
and 3, the researchers obtained a general picture of how markers 
of sex and race intersect with activities involving technology. 

Vignette analysis (Maxwell 1996) was used to investigate the 
question about occasions and related conditions in which the 
girls demonstrated a motivation to engage in science and tech- 
nology learning. In vignette analysis, salient events that bear on 
the research question are excerpted from the data texts, simpli- 
fied, related to one another, and retold in the form of short sto- 
ries called vignettes. The objective is to develop a description 
of the context of the activity that links the data to a "coherent 
whole" (Van Maanen 1988). 

The Eisenhart and Edwards (2004) analysis revealed the after- 
school program to be a hybrid space between the formal space of 
the school classroom and the informal space of students' social 
lives outside school, in which participants are free to move out- 
side the roles expected of them and participation can be more 
symmetrical (Gutierrez, Rymes, and Larson 1995). This created 
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opportunities for the girls to make connections between science 
and their own lives and values, connections that Eisenhart and 
Edwards found helped sustain the girls' engagement. They also 
found that the absence of such connections made it difficult for 
the girls to sustain engagement. 

Metacognition in an Amusement Park Physics Program 

Levels of Analysis Individual, group 

Valued Outcomes Collaboration, independent inquiry 

Methods Interpretive case study, semistructured interviews, 

pre- and post-experimental assessments 

Anderson and Nashon (2007) studied an informal learn- 
ing program that was organized to support school classroom 
work. The program took high school juniors and seniors who 
were enrolled in physics classes to an amusement park, where 
they were divided into teams to assess the kinematics of three 
rides using accelerometers, stopwatches, and protractors. This 
amusement park activity was combined with a classroom assign- 
ment in which the students, working in their original teams, 
were given prompts to encourage them to develop explanations, 
arguments, and models of the physics principles in the rides. 

Grounded in an interpretive case study approach (Gallagher 
and Tobin 1991; Merriam 1998; State 1995), the study aimed to 
assess the students' metacognitive abilities (individually and as 
groups) and how these abilities influenced the development of 
the students' conceptual understanding of kinematics. 

Although the authors describe metacognition and learning as 
idiosyncratic and dynamic processes, this research is centered on 
the psychological construct of metacognition, measured through 
a 53-item baseline questionnaire the researchers developed that 
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allowed them to profile each student according to the constitu- 
ent components of metacognition: awareness, control, evalu- 
ation, planning, monitoring, and self-efficacy. These baseline 
metacognitive levels became an important lens through which 
the researchers interpreted individual students' discursive inter- 
actions in subsequent activities, which in turn contributed to 
the validation of the baseline instrument. 

After taking the baseline survey, the 11th and 12th graders 
in the study visited and conducted their experiments in small 
groups at the amusement park. During the visit, the research 
team developed field notes, video recordings of several small- 
group interactions, and individual audio recordings of conversa- 
tions of the students participating in the in-depth case studies. 

The case study students were given copies of their personal 
conversations and asked to listen to them in advance of group 
interviews about their metacognition and learning. These stu- 
dents participated in one group interview after the field trip, 
some in-class activities related to the physics explored on the 
field trip, and a second group interview after the in-class activi- 
ties. Audio and video recordings were made of all these events. 

In their analysis, Anderson and Nashon (2007) interpret stu- 
dents' discussions in learning activities and interviews through 
the lens of their particular metacognitive profiles. The research- 
ers illustrate how these profiles interact with students' circum- 
stances, dynamically influencing social processes within their 
small peer groups and concurrently influencing their individual 
and collective formation of physics concepts. 

Museum-Based Programs and Projects 

In this section we describe several museum-based, informal 
learning projects. Informal science institutions such as science 
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and technology museums, aquariums, and zoos have long 
offered specific educational programs for children and youth as 
well as educationally oriented general exhibitions for the public 
of all ages. Research studies on learning in such institutions have 
focused on individual and group (especially family) learning and 
engagement with exhibits as well as learning in more specialized 
programs, including various youth-oriented ones in physical 
and online spaces (Herr-Stephenson et al. 2011). Because these 
settings do not especially afford opportunities for testing visi- 
tors' knowledge before and after, studies have often developed 
other means for assessing learning and other criteria for valued 
outcomes. 

GIVE: Facilitating Group Inquiry in Science Museums 
Levels of Analysis Individual, group, project 
Valued Outcomes independent inquiry, collaboration 
Methods Experimental design, video analysis 

The GIVE (Group Inquiry by Visitors at Exhibits) research 
project was launched at the San Francisco Exploratorium, a lead- 
ing informal science learning institution (Gutwill and Allen 
2010). The project aimed to answer the following questions: 

• Can intergenerational groups of museum visitors, such as fam- 
ilies, be coached by the museum staff to learn a set of inquiry 
skills that they can use on their own at novel exhibits or even 
experiences beyond their visit? 

• How does such an intervention affect visitors' inquiry behav- 
iors at a novel exhibition? Does it support them to explore the 
exhibit more deeply? 

• What properties of the staff-mediated intervention seem cen- 
tral to its design, and what is the evidence for them? 
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To probe these questions, the GIVE team developed two 
inquiry games based on the following learning science-derived 
design principles: building on prior knowledge; teaching through 
modeling, scaffolding, and fading; identifying skills explicitly; 
supporting metacognition; supporting collaboration; making 
the activity intrinsically motivating; minimizing the cognitive 
load; supporting visitors' learning agendas; and supporting indi- 
vidual spontaneity. (The last four are specific to informal learn- 
ing design.) 

Because of constraints in the museum setting, the games the 
researchers developed targeted only the following two among 
the full range of inquiry skills: 

• Proposing actions Asking a question or making a plan at the 
beginning of an investigation. 

• Interpreting results Making observations, interpretations, or 
explanations during or after an investigation 

The researchers chose these skills because they were simple 
enough to learn quickly and were not ones already commonly 
observed in museum visitors' behaviors. The researchers also 
viewed these two skills as possible gateways to other inquiry 
skills, such as questioning, predicting, analyzing, and explain- 
ing. Along with experimentation, an activity that visitors engage 
in spontaneously at interactive exhibits, proposing actions and 
interpreting results form a complete, if simplified, version of the 
inquiry cycle as it is typically presented in school curricula and 
characterized by science researchers. 

After iteratively refining the games, the GIVE team conducted 
an experimental study comparing the two game conditions with 
two control conditions. They studied 50 families in each condi- 
tion, for a total of 200 families, to determine whether the games 


Review of the Literature 


45 


promoted inquiry behavior at exhibits. Families in the treat- 
ment groups learned how to play the games with supports at 
two or more exhibits and then, as a post-test, were asked to use 
a final exhibit playing the game they had learned. Afterward, 
the research team conducted interviews with one adult and one 
child per family, selected at random. Researchers also collected 
demographic data on each participant. 

The GIVE team video-recorded all families at their first and 
last exhibits and coded the participants' behavior according to 
the following codes: engagement, proposing actions, interpret- 
ing results, collaborative explanations, and coherent investiga- 
tions. They also coded the data for the correctness of the science 
content. The results showed that the inquiry games succeeded 
in improving the participants' inquiry. The more structured and 
collaborative of the two games had the strongest effect, leading 
to longer engagements with the post-test exhibit, more abstract 
interpretations, and more collaborative and coherent inquiry 
investigations than controls. Based on their qualitative analy- 
ses, researchers attributed the greater success of this game to its 
inclusion of all family members in collaboration and its sup- 
port of their explicit articulation of their interpretations of the 
exhibits. 

TOBTOT: Assessing Museum Learner Talk over Time 
Levels of Analysis Individual, group 
Valued Outcomes Collaboration, independent inquiry 
Methods Discourse analysis, thematic analysis 

This study (Ash et al. 2007) was designed to explore dialogue- 
based methodological approaches, in particular the authors' 
Tool for Observing Biological Talk over Time (TOBTOT). The 
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tool was used to analyze scientific sense making in biological 
talk in out-of-school settings and track the language people actu- 
ally use in discussing life science themes over time. 

The researchers' methodological concerns stem from several 
theoretical traditions, exemplified by the works of Lev Vygotsky, 
Michael ffalliday, James Wertsch, Mikhail Bakhtin, Jay Lemke, 
and others that take a sociocultural view of conversation as both 
structured and dialogically emergent (Mahn and John-Steiner 
2013; Wells and Claxton 2002). Their applications of the TOB- 
TOT framework focused on testing ways to track thematic pat- 
terns in order to analyze, in both quantitative and qualitative 
terms, the development of scientific content in dialogic inter- 
action. Ash et al. (2007) focus on the methodological tensions 
entailed by working along the dimensions that encompass con- 
tent and process, everyday language and canonical scientific dis- 
course, macro- and microlevels of analysis, and qualitative and 
quantitative representations of data. 

During a three-year period, the research team audio- and 
video-recorded 20 Spanish-speaking and English-speaking fam- 
ily groups as they interacted for 20 to 80 minutes at exhibits in a 
marine biology center in northern California. The families were 
recruited from a local ffead Start center, and each included at 
least one parent and two preschool- or elementary school-age 
children. The researchers also conducted interviews with family 
members after the data collection sessions at exhibits, asking for 
reflections on selected video clips of their naturalistic interac- 
tions as a member check. 

The analytic tool used, the TOBTOT, consists of three super- 
ordinate thematic categories for coding dialogue related to 
biological phenomena: Staying Alive, Characterizing, and Eco- 
logical Interdependence. Staying Alive roughly corresponds to 
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the concepts and discourse traditionally associated with the dis- 
cipline of biology. Characterizing is the TOBTOT category for 
talk related to labeling or locating an animal. Ecological Inter- 
dependence categorizes talk thematically related to habitats, 
human impact, and, less typically for life-science disciplines, 
feelings and aesthetics (e.g., expressions of amazement or per- 
sonal attraction to an organism). 

On a macroanalytic level, TOBTOT's categorization scheme 
allowed the researchers to illustrate and provide graphic repre- 
sentations of how families discursively engage with abstract bio- 
logical content differently from one another, as well as across 
exhibits and over time. The research team reported TOBTOT- 
based evidence that suggested several interesting findings and 
noted their implications. 

First, the naturalistic family dialogue centered on Character- 
izing and Ecological Interdependence more than Staying Alive 
talk, emphasizing the importance in everyday talk — and science 
learning in informal settings — of becoming familiar with and 
feeling for living things. 

Second, Characterizing and Ecological Interdependence 
tended to occur before Staying Alive talk at an exhibit, suggest- 
ing that "naming, using prior knowledge, and making personal 
connections must occur before the formal science can begin” 
(Ash et al. 2007, 1589). 

Third, particular themes occurred as leitmotifs that changed 
over time, illustrating that "questions (e.g., about feeding) can 
permeate time and context, often acting as a central core of dia- 
logic negotiation." Fourth, the touch tank generated the most 
family talk, prompted the researchers "to investigate the special 
role of living things as mediational means in dialogic activity” 
(Ash et al. 2007, 1591). 
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Finally, the researchers also illustrated the ways in which the 
TOBTOT could be used to analyze and document differences 
between families in their patterns of discursive interaction at 
exhibits. 

The TOBTOT proved useful in helping researchers demon- 
strate science content in everyday language, largely because it 
accommodates talk in the Characterizing and Ecological Inter- 
dependence categories. Ash et al. (2007) point out that "the 
general observations these families made were predicated on sci- 
entific principles" that are part of "cultural scripts based upon 
common scientific understandings" (1594). 

However, they also note the difficulties they had in deter- 
mining whether some instances of everyday talk were scientific 
or not. They further noted that although their data indicated 
that the "families used multiple resources to talk and act in new 
ways, and the use of these resources were dynamic, social, and 
discontinuous, not linear, direct or clear" (Ash et al. 2007, 1595), 
the TOBTOT was not able to capture the nuances of the families' 
dialogic interactions and the scientific insights they supported. 

The authors also reported that the use of the TOBTOT did 
not resolve tensions related to particularistic treatment of con- 
versational data or readily facilitate conversational segmenting. 
Therefore, the research team struggled with traditional stan- 
dards of reliability and the quantitative representation of quali- 
tative data. 

The researchers conclude that the TOBTOT is a powerful 
tool for creating abstract interpretations and representations of 
actual lived experience in order to answer relatively simple ques- 
tions. The TOBTOT did not help answer the hardest questions, 
especially regarding the cultural resources people use to make 
sense of and attach value to natural phenomena in their world. 
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The Zydeco Tool: Inquiry in Museums and Classrooms 

Levels of Analysis Individual, group 

Outcomes Independent inquiry, collaboration 

Methods Analysis of computer system logs, ethnographic field 

notes, interviews 

Chris Quintana and his colleagues (2010) at the University 
of Michigan's Center for Highly Interactive Classrooms, Curri- 
cula, and Computing in Education have developed projects that 
combine tools and corresponding curricula for helping students 
engage in scientific inquiry in activities that combine museum 
and classroom activities. 

The Zydeco project is one example of this work. It consists of 
three components: (1) a Web site where students develop and 
access study questions, hypotheses, and data, (2) a data collec- 
tion and annotation component implemented as an iPhone or 
iPod app, and (3) an explanation component implemented as an 
iPad app to support students' visualization and use of the data 
they collected for constructing explanations relevant to the phe- 
nomena they are collaboratively exploring (Kuhn et al. 2012). 
The system was designed for what has been termed nomadic 
inquiry : structured inquiry across a variety of settings, including 
classrooms, museums, the outdoors, and in homes (Hsi 2003). 
The Zydeco system is designed to help learners easily transition 
from one setting to the next, including transitions between vir- 
tual (handheld devices) and physical (e.g., museum exhibits) 
contexts. 

Kuhn et al. (2012) examined the role that annotations played 
in how students searched and evaluated data and how students 
identify and select the evidence that they use in their explana- 
tions. They followed a group of middle school students, grouped 
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into pairs, who used the Zydeco system to collect data at a natu- 
ral history museum for use in a classroom-based project. This 
project involved the students using the explanation construc- 
tion component to examine their data along with all the data 
collected by other students during the museum visit, and using 
it to formulate their own explanations. Kuhn et al. found that 
the annotations made by the students did support inquiry activi- 
ties, including data interpretation, identification, and search. 

The documentary and assessment methods used for this study 
were a combination of interview data, field notes, and usage logs 
taken from the Zydeco system. Six pairs of the 54 students fol- 
lowed were given approximately five-minute, daily in-process 
interviews using a semistructured interview protocol while they 
used a tablet to do the explanation exercise. 

The log data was studied together with researcher-generated 
field notes of the classroom activity. This included data collected 
by the students in the museum, which allowed the researchers 
to study the characteristics of the data, including the type of 
data (e.g., photos, audio notes, photo-audio note pairs) and the 
accuracy of the annotation titles, tags, and audio notes. A rubric 
was used to grade the students' final explanations. 

WINS: Career Support in a Science Museum After-School Program 
Levels of Analysis Individual, group 
Valued Outcomes Social resources 

Methods Longitudinal, survey, and interview-based study 

Fadigan and Hammrich (2004) conducted a study to describe 
the educational and career trajectories of high school girls 
who participated in a structured museum-based natural sci- 
ence enrichment program know as WINS (Women in Natural 
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Sciences). Developed for academically talented women from 
urban, low-income, single-parent families, the program's aim 
was to give participants "the information, encouragement, and 
confidence they need to consider pursuing careers in the natural 
sciences, to make informed decisions, and to shape their own 
futures" (840). 

In addition to providing participants and their families with 
free access and transportation to the museum, the yearlong pro- 
gram put the participants in summer classes on environmental 
science, sent them on field trips to local parks, and allowed them 
to have extended stays at an environmental education center. 
During the academic year, students met at the museum once a 
week, attended monthly field trips to other science education 
institutions, and met scientists. 

In order to describe the overall character of the educational 
and career trajectories for the program's participants, Fadigan 
and Hammrich (2004) selected a sample of former participants, 
examined the application materials submitted by these par- 
ticipants, and from these materials identified the educational 
and career trajectories that these participants had imagined for 
themselves. Surveys designed to capture details about the par- 
ticipants' actual educational and career trajectories (high school 
to the present) were mailed to the students in this sample. 

After the analysis of the surveys, a representative subsample 
of the participants was invited to come in for interviews. The 
interview questions were developed to assess how and why these 
former participants had pursued the educational and career 
paths that they did, and to identify what if any effect WINS had 
had on the development of these paths. The design of the study 
and study instruments was based on Eccles (1994), a model of 
achievement-related choices that posits that an individual's 
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choices about what career paths to pursue are constrained by 
the knowledge they possess about possible options, perceptions 
of gender-role positioning, and perceptions of the value and 
importance of the options. 

The types of findings on valued outcomes that this approach 
permitted included the following: 

• 93 percent of the participants had enrolled in a college pro- 
gram after completing high school. 

• Careers in medical or health-related fields, followed by careers 
in STEM, emerged as the highest-ranking career paths, with 
about 20 percent of respondents pursuing careers in each of 
these areas. 

The majority of participants named several WINS elements as 
influencing their educational and career decisions: having staff 
to talk to, learning job skills, and having the museum as a safe 
place to go. 

Computer-Based and Online Activities 

Computers make it possible to keep track of what a person does 
on them minute by minute. Whether a person is using a com- 
puter simulation, an online multiplayer virtual world, or a sim- 
ple text editor, it is now possible to comb through log files or 
through lines of code to generate records of every action a per- 
son takes with these kinds of software. This information, when 
interpreted using the relevant conceptual frameworks, can reveal 
something about what and how a person is or is not learning. 

A key advantage of this approach is that it permits learn- 
ing assessment without disrupting the flow of learning activi- 
ties. Another advantage is that these kinds of computer-based 
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assessments can also be accomplished in real time, which allows 
for feedback to the learner in real time also, but again, not in 
a manner that makes the learners aware that they are being 
assessed. 

In this section we describe examples of approaches to con- 
ducting this kind of embedded assessment: stealth assess- 
ment in commercially available games, Web and project 
analysis in Scratch programming, and learning analytics meth- 
ods to assess programming and learning in heavily documented 
environments. 

Stealth Assessment in Commercially Available Games 
Level of Analysis Individual 
Valued Outcome Independent inquiry 
Method Evidence-centered design 

Stealth assessment refers to a methodology for integrating 
learning assessments into the structure of computer game-based 
educational activities in such a way that relevant data about 
the learning process can be gathered in real time without dis- 
rupting student engagement in the activity. The main assump- 
tions underlying stealth assessment research are that learning by 
active participation in the game improves outcomes, different 
learner attributes may be identified during game play, strengths 
and weaknesses of the learner may be addressed to improve 
learning, and formative feedback can be used to further support 
student learning (Gee 2003; Gee and Shaffer 2010; Shute 2007, 
2008; Shute, Hansen, and Almond 2008; Shute and Ventura 
2013; Squire 2006). 

In addition to providing a rich context in which to study 
learning, video games are a central focus of stealth assessment 
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Box 1 

Evidence-Centered Design 


Key projects reviewed here employ variants of a conceptual 
framework for creating learning assessment procedures known as 
evidence-centered design (ECD). ECD makes formal and explicit 
both the process of designing assessment procedures and the pro- 
cedures themselves. It provides language and concepts for sys- 
tematically developing interlinked models of the design, analysis, 
deployment, and measurement aspects of learning assessment 
(Behrens et al. 2004; Mislevy, Almond, and Lucas 2004; Mislevy 
and Haertel 2007). 

The process of developing these assessments involves analysis 
of the target-learning domain in order to establish domain repre- 
sentations, categories, and features relevant to addressing assess- 
ment goals. This establishes what is called the competency model 
for a specific assessment and answers the question, What collec- 
tion of knowledge and skills should be assessed? ECD also requires 
specifying the kinds of student behaviors and actions that will 
constitute evidence of learning, the evidence model, answering the 
question, What behaviors or performances should reveal those 
constructs? Finally, these two models are connected by a concep- 
tualization of how the assessment will be implemented, the task 
model, which answers, What tasks should elicit those behaviors 
that constitute the evidence? 

Whereas initial efforts to apply ECD centered on the develop- 
ment of assessments for use in formal educational settings, more 
recent efforts have applied the framework to activities and set- 
tings that fall within the scope of this report. We describe three 
examples: (1) so-called stealth assessment of player learning in 
commercially available first-person, role playing video games; (2) 
networked analyses for assessing learning in epistemic games; and 
(3) learning assessment of an innovative, large-scale implementa- 
tion of a game-design curriculum. 
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because, as activities instantiated through computer platforms, 
they afford convenient and reliable tools for continuously 
recording and analyzing student performance data in real time. 
Gathering this documentation is not generally a problem; how- 
ever, making sense of the potentially massive amounts of data 
generated requires a principled conceptual framework. 

Furthermore, once this data has been interpreted, there is the 
necessary and complex task of feeding this information back 
into the educational activity. For example, in computer game- 
based environments, this information can be used to support 
real-time adaptive adjustments to the game in ways that pro- 
mote and maintain player learning. On a long-term basis, it is 
also important to communicate to both students and teachers 
what was learned from these assessments so that the learning 
activity can be restructured for improved and sustained learning. 

In the past several years, Valerie Shute and her colleagues 
have pioneered stealth assessment methods (Shute 2011; Shute 
et al. 2009). In this section we draw on this corpus of work to 
explain the application of the evidence-centered design (ECD) 
framework, described in box 1, in assessing learning in video 
games. Games that incorporate stealth assessment must elicit 
behaviors that provide evidence of the skills and knowledge 
being assessed, and they must afford principled interpretations 
of the evidence in terms of the aims of the assessment. As noted, 
ECD guides the researcher in developing a framework for col- 
lecting and analyzing data in terms of three interrelated models, 
the first of which is the competency model, which names the 
sets of knowledge and skills on which assessment inferences are 
to be based. We focus here on an example described by Shute 
et al. (2009), in which the researchers studied players' creative 
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problem-solving abilities in Oblivion, a commercially available 
first-person, role-playing video game (Bethesda Softworks 2006). 

Figure 4 shows a simplified competency model developed by 
Shute and her colleagues for modeling creative problem solving 
in the game. The model defines and situates creative problem 
solving in a restricted range of educationally relevant competen- 
cies that one could assess in Oblivion game play. Student-specific 
instantiations of competency models are termed student models. 
The values reported in these models, like profiles or report cards, 
denote assessor beliefs about the student's level on each of the 
variables in the competency model. 
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Figure 4 

Simplified competency model for the commercially available role-play- 
ing game Oblivion. The areas shaded in gray represent the variables used 
to assess creative problem solving. From Shute (2011). 



Review of the Literature 


57 


Shute et al. (2009) operationally define creative problem solv- 
ing as a contextually specific "mental process of creating a solu- 
tion to a problem ... in which the solution is independently 
created rather than learned with assistance" (309). By emphasiz- 
ing problem solving according to this definition, assessment is 
focused on the novelty and efficiency of approaches to accom- 
plishing tasks in the game. 

The evidence model here defines the line of argument 
describing how and why player performances in a given task sit- 
uation constitute evidence about competency model variables. 
Two questions guide the development of the evidence model: 
What behaviors or performances reveal targeted competencies? 
What is the functional (or statistical) connection between those 
behaviors and the competency model variable(s)? 

Looking specifically at video games like Oblivion, connections 
between game observables and their corresponding competen- 
cies require that the evidence model include (1) scoring rules for 
extracting observables from players' game play indicators found 
in log files, (2) the observables (i.e., scored data), and (3) mea- 
surement rules for accumulating evidence from the observables, 
which are then used to update the student model variables. 

In the present example (creative problem solving), novelty 
and efficiency link actions in the game with competencies 
related to creative problem solving. Efficiency is defined in terms 
of the quality and quantity of steps taken to solve the problem; 
novelty is defined in terms of the frequency with which certain 
actions are taken (i.e., low-frequency actions are more novel). 

The task model here serves as a framework for conceptualiz- 
ing game-level situations that elicit player performances deemed 
to provide evidence of competency-related knowledge. Among 
other things, the model specifies what the player will be asked to 
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do and what kinds of responses are allowed. In the specific case of 
stealth assessment, Shute et al. (2009) refer to the task model as 
the action model in order to highlight the fact that the modeling is 
focused on defining a player's action sequences and each action's 
indicators of success. Actions are defined as anything a player 
does to solve a particular problem contained within a scene in the 
game. Indicators are explicitly linked to each action, and these 
indicators can in turn be measured from the player's log file. 

The Scratch Community Web Site 
Levels of Analysis Individual, group 
Valued Outcomes Collaboration, quality of product 
Method Data mining (Web site) 

Another area in which the documentation power of comput- 
ers is being harnessed to assess student learning is programming 
and digital media production. We return here to the example 
of the Scratch programming platform (see Computer Clubhouse 
discussion above). Maloney et al. (2008) set out to describe 
which programming concepts the students (8 to 18 years old) 
at a Computer Clubhouse used in their projects, how they used 
these concepts, and the extent to which the community of the 
clubhouse as a whole increased its collective knowledge of com- 
puter programming over time. 

Documentation consisted of both qualitative and quantita- 
tive measures. Scratch project summary files were exported and 
collected. These contained information about the number and 
kind of programming commands used along with records of the 
frequencies of stacks, sounds, and costumes used in the project. 
In addition, weekly field notes of Scratch activities were gathered 
by university students, who attended the Computer Clubhouse 
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as both researchers and Scratch mentors. Finally, interviews with 
clubhouse members were conducted to document their impres- 
sions of Scratch and to obtain histories of their programming 
experience. 

The summary files were analyzed for the frequency and dis- 
tribution of Scratch commands in the corpus of projects stud- 
ied. This allowed Maloney et al. (2008) to develop a profile of 
the programming commands in use over an 18-month period 
by clubhouse members as a group. The researchers compared the 
frequency of programming commands included in projects from 
one year to the next, and they analyzed the difference in per- 
centages of projects containing targeted programming concepts. 
Significant gains were observed for the use of five out of seven 
targeted programming concepts. 

One finding that highlights the importance of the relation- 
ship between the social dynamics at the clubhouse and what 
individuals in these settings do with these technologies is that 
the particular programming commands that were frequently 
used were commands that were important for creating programs 
that were seen as valuable by members. For example, in this par- 
ticular clubhouse, games and animation were popular among 
the students. This in turn was reflected in the fact that many of 
the Scratch programs produced by the students included com- 
mands relevant for games and animations. 

The university mentors selected to assist clubhouse mem- 
bers with Scratch knew little to nothing about programming. 
This was by design; with inexperienced programmers as men- 
tors, clubhouse members were expected to feel more empowered 
in their learning, even bolstering their programming expertise 
in situations when they would be asked to help mentors learn 
about Scratch (Kafai et al. 2007). In their study, Maloney et al. 
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(2008) argued that given the minimal role played by mentors at 
the clubhouse in terms of any direct instruction in Scratch, the 
Scratch environment itself was most likely responsible for the 
learning that resulted. 

In order to assess the power of the Scratch environment as 
a stand-alone programming learning environment, Dahotre, 
Zhang, and Scaffidi (2010) studied Scratch programs created by 
youths who did not attend Computer Clubhouse. To do this they 
used a "screen scraper" program to download from the Scratch 
community Web site 100 random animation projects, their cor- 
responding computer code, and user comments. The HTML for 
the repository Web page was also downloaded in order to assess 
usage statistics (i.e., counts of views, comments, downloads, and 
visitor remixes) for each animation project. These projects were 
then assessed in terms of three learning goals envisioned by the 
developers as being facilitated by Scratch: learning of technical 
programming skills, social skills for collaboration, and socio- 
technical remixing skills related to adapting existing programs 
and community resources to produce new programs. 

The projects that Dahotre et al. (2010) evaluated from the 
community Web site included key programming primitives at 
the same levels as the projects that were developed under par- 
tial supervision of undergraduate mentors (e.g., Maloney et al. 
2008). In order to assess these projects against the general popu- 
lation, Dahotre et al. drew on findings from methodologically 
similar studies based on other repositories of end-user program- 
ming code (e.g., spreadsheet programming and macros). They 
found that the Scratch programs showed comparable or higher 
rates of use of programming primitives. 

To assess collaborative and remixing skills, Dahotre et al. 
(2010) studied the comments posted by users to projects on the 
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Web site and project download statistics, respectively. Although 
the majority of comments posted were helpful critiques, they 
did not indicate that students were actively collaborating with 
one another on projects. Regarding remixes, the authors found 
that the relatively low levels of remix activity were comparable 
to those found in the general population. 

More recently, Scaffidi and Chambers (2012) extended the 
work of the above studies to examine the extent to which the 
Scratch environment, via the Web community, facilitated the 
development of social skills or elementary programming skills. 
To this end they studied data from more than 1,000 Scratch proj- 
ects posted to the community Web site by 250 randomly selected 
users. Drawing on the existing literature, Scaffidi and Chambers 
adapted four models of end users to create a framework for inves- 
tigating skill progression in Scratch. Two models addressed ques- 
tions of programming skill related to the use of programming 
primitives, one examined social skills, and the last concerned 
programming efficiency based on the number of lines of code 
produced and the amount of time taken to produce them. 

Learning Analytics in Computer Programming 

Level of Analysis Individual 

Valued Outcome Independent inquiry 

Methods Learning analytics, data mining (programming code) 

Paolo Blikstein and his colleagues at Stanford's Transfor- 
mative Learning Technologies Lab have developed a number 
of automated learning analytics approaches for making quan- 
titative assessments of the kinds of open-ended, nonscripted 
learning activities that, as this survey of the literature demon- 
strates, have been previously largely qualitative. One example 
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of this work concerns the assessment of students' programming 
know-how. 

Blikstein (2011) examined assignments completed by nine 
sophomores in an undergraduate programming class. The stu- 
dents were asked to choose a specific scientific phenomenon 
and write a program that modeled the phenomenon. Student 
programming logs were collected using the NetLogo (Wilensky 
1999) programming environment. NetLogo is capable of logging 
to an XML file such user actions as key presses, button clicks, 
and changes in programming code. 

Blikstein (2011) developed a special configuration file for 
specifying and logging the targeted programming actions. This 
file was distributed to the students, who were given instruc- 
tions for enabling it so that log files could be collected and then 
processed, coded, and analyzed. The analysis focused on iden- 
tifying the coding strategies that the students employed. Once 
identified, these strategies were combined with survey infor- 
mation about the students' previous programming knowledge 
to determine coding profiles for each student (e.g., copy-and- 
pasters, self-sufficients, or mixed mode). The broader aim here 
was to draw on the students' observed programming patterns to 
improve the design of teaching and support materials and strate- 
gies, as well as to identify critical points in the process of writing 
software applications where support interventions might best be 
included. 

Ethnographic Studies of Online and Gaming Communities 

The next two studies represent a useful approach to the research 
and assessment of online communities in which the researcher 
acts as a member of the community or otherwise studies the 
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online community as an ethnographer might study a traditional 
face-to-face community. Insofar as informal learning increas- 
ingly occurs today in such online communities, the methods 
of documentation and assessment developed within them are 
important for the field. 

Lineage, World of Warcraft, and Second Life 
Levels of Analysis Individual, group 

Valued Outcomes Collaboration, independent inquiry, social 
resources 

Method Cognitive ethnography 

Steinkuehler (2008) has described a program of research for 
studying learning in massively multiplayer online role-playing 
games (MMORPGs) such as World of Warcraft, as well as non- 
game online virtual worlds like Second Life. The first phase of 
the research involved conducting a cognitive ethnography 
(Hutchins 1995) of the game and its community of players with 
the aim of identifying the forms of cognition and learning that 
lead to successful game playing. 

Steinkuehler (2008) studied the fantasy-based MMORPG 
Lineage through 28 months of participant observation. This 
involved the study and documentation of naturally occurring 
game playing through video and field notes, interviews with 
players, and the archiving and analysis of game-related player 
communications (e.g., emails, chat room and instant-message 
conversations, and discussion-board posts) and community 
documents (e.g., fan Web sites and community-authored game 
fictions). 

Five broad categories of social and intellectual practices 
that characterize successful MMORPG play were identified: 
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(1) collaborative problem solving, (2) digital media literacy 
practices, (3) informal scientific and mathematical reasoning, 
(4) computational literacy (e.g., "modding"), and (5) cultural 
mechanisms for learning, including reciprocal apprenticeships 
(Steinkuehler 2004; Steinkuehler and Williams 2009) and collec- 
tive intelligence (Steinkuehler 2006). See Steinkuehler (2005) for 
a full survey of the practices identified. 

Note that for players of MMORPGs, there are normally two 
intersecting online communities: the in-game community of 
interactions with other players during actual game playing, and 
the out-of-game online guilds and associations in which infor- 
mation about the game and game planning and strategies are 
developed and exchanged. 

In the second phase of research, Steinkuehler (2008) drew 
on the social and intellectual practices identified in the first 
phase and applied them as guiding themes for conducting tar- 
geted empirical investigations of World of Warcraft and Second 
Life. These studies yielded observations about collaborative 
problem solving and the multiple literacies that players develop 
as a result of participation in the game and the game commu- 
nity. Steinkuehler notes important parallels between the kind 
of collaborative problem solving required in games like World 
of Warcraft (e.g., group-organized "raiding") and those required 
in contemporary workplace environments. She also reports on 
the high degree of motivation that players show for developing 
multiple literacies, including computer literacies (e.g., creating 
"mods") and practices that overlap with more traditional litera- 
cies (e.g., writing prose in the form of fan fiction). 

For an ethnographic study focused on collaborative problem 
solving in World of Warcraft, see Chen (2009). 
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Whyville 

Levels of Analysis Individual, group 

Valued Outcomes Collaboration, independent inquiry, social 
resources 

Method Connective ethnography 

Online virtual worlds have become pervasive as educational 
tools, both in and out of school. Regardless of the setting, imple- 
menting these worlds and assessing their potential learning ben- 
efits remains a challenge. A key question raised by environments 
in which learners inhabit and travel between virtual and physical 
spaces is how this learning can be traced. To examine this ques- 
tion, Fields and Kafai (2007) conducted a connective ethnography 
of a summer after-school program in which fourth to sixth grad- 
ers used an online virtual world, Whyville (see also Kafai 2010). 

Citing work by Hines (2000) and Leander and McKim (2003), 
Fields and Kafai (2007) define connective ethnography as an 
approach that focuses on examining how everyday practices 
work to either segregate or blend social spaces. These processes 
are studied by tracing the flow of objects, text, and bodies and 
how this flow contributes to the construction of boundaries 
within and between virtual and physical spaces. 

Within this framework, Fields and Kafai (2007) adopted a 
strategy of identifying forms of insider knowledge about the 
game that would afford insights into the kinds of learning that 
players experience within and between the virtual world and the 
physical space of the after-school center. This knowledge had to 
be traceable, discoverable by players through trial and error, and 
observable in and out of the virtual world. Understanding the 
particular insider knowledge that the researchers studied — tele- 
porting — requires a brief introduction to the basics of the game. 
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Whyville is a multiuser virtual environment for 8- to 16-year- 
olds in which the users are positioned to engage in a variety of 
science-themed activities. A player who successfully completes 
an activity earns virtual currency, which can then be used to buy 
and design features of the player's avatar as well as other goods 
useful in the world of the game. Social interaction in the game 
is mediated through "ymailing" (email within the Whyville uni- 
verse) and avatar-to-avatar chat. Avatars interact in more than 
30 different settings, each of which can be accessed through a 
pull-down menu (see figure 5). 

Not all the locations that players can visit are visible in the 
menu (e.g., the Moon, Jupiter, and Saturn are not). These secret 
sites are accessible only by teleporting, an action that is accom- 
plished by typing "teleport [place]" in the chat bubble above the 
avatar's head. Teleporting is not observable in the game because 
the avatars are transported to the new location before the typed 
commands are displayed, which prevents other players from 
knowing where and how the avatar disappeared. The locations of 
these secret sites and the ways of getting there are bits of informa- 
tion that can be obtained only through conversation with other 
players or from Whyville "cheat" Web sites. Consequently, these 
secret locations gain a special status among Whyville insiders as 
social hangouts free of newbies (new, relatively naive players). 

To document learning over the course of the nine-week pro- 
gram, Fields and Kafai (2007) adopted a data multistreaming 
approach. Daily field notes were taken to document the overall 
activity at the after-school center, and video was used to capture 
interactions of small student groups as they played while clus- 
tered around a computer. Online tracking data was recorded for 
players' chats and the locations of their avatar throughout the 
game. Finally, individual postactivity interviews of the partici- 
pants were conducted. 
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Figure 5 

From top to bottom: the pull-down destination menu, and two of the 
30 destinations where players can visit and interact — the beach and the 
moon. From Kafai and Fields (2013). Image © Numedon, Inc. 2013. 
Reprinted with permission. 
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All text documentation was combed for instances in which 
teleportation was mentioned. This allowed for a mapping of 
each player's history of learning and using the teleportation 
function. For example, chat transcripts for each student were 
studied to identify the first time that students discussed telepor- 
tation in the game and when they first used the teleportation 
function. This information was in turn triangulated with the 
field notes and video data to paint a more detailed picture of the 
contexts in which players learned and implemented the telepor- 
tation function. 

Fields and Kafai (2007) found that for the student players 
there was a seamless integration between online and offline 
interactions. What happened in Whyville was just as important 
as what happened in the after-school center. For this reason, 
Fields and Kafai propose referring to activity in these worlds as 
"synthetic play," as in the synthesis of online and offline worlds. 

Additional Studies of Interest 

In this section we include detailed accounts of two additional 
research projects that have developed relatively sophisticated 
approaches to the documentation and assessment of settings in 
which digital games provide key learning opportunities. In each 
case, these projects draw on and elaborate the ECD framework 
discussed earlier. 

Digital Zoo: Epistemic Games and Epistemography 
Levels of Analysis Individual, group 

Valued Outcomes Independent inquiry, know-how, collabora- 
tion, social resources 
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Methods Evidence-centered design, epistemography, epistemic 
network analysis 

The research we describe here is drawn from the work of 
the multi-institutional Epistemic Games Research Group led 
by David Williamson Shaffer. The group includes researchers, 
faculty, staff, and students from the University of Wisconsin at 
Madison, the University of Maryland, the University of Mem- 
phis, the Massachusetts Audubon Society, the Danish School of 
Education, and the Open Universiteit Nederland. 

Shaffer et al. (2009) study digital learning systems: systems 
constituted by a theory of learning and corresponding methods 
of assessment, which are linked into an evidence-based, digital 
intervention. The digital learning systems that Shaffer and his 
colleagues developed and studied are known as epistemic games. 

Epistemic games are activities through which students learn 
the knowledge, skills, practices, and values of a particular profes- 
sional domain. These activities are mediated through a variety of 
digital technologies, including virtual-world simulations of the 
tasks typically engaged in by professionals in the target field, 
as well as the computer applications used by the professionals 
in their everyday work. Players take on the role of apprentices 
in such professions as city planning, science journalism, and 
engineering. 

The learners are guided through these worlds by the deliber- 
ate inclusion of game tasks that ask the learners to reflect on 
their work. This can involve interactions with peers and/or with 
actual professionals in the relevant fields. These interactions are 
designed by the game creators and researchers to model the kind 
of mentoring that professionals experience in their training and 
socialization. 


70 


Review of the Literature 


An example of an epistemic game is Digital Zoo. In Digital 
Zoo, players adopt the role of biomechanical engineers who are 
given the task of using specially designed software to create vir- 
tual objects and creatures for an animated film. Design specifica- 
tions for the desired appearance and movement of these objects 
and creatures are delivered to the players, who are then tasked 
with creating these objects. Through the design elements of the 
activity as a whole and through the participation of knowledge- 
able mentors, the players are guided in following the roles and 
rules of engineering design based on research examining how 
real engineers in training learn to design. 

The epistemic frames hypothesis (Shaffer 2006) is a theory 
of learning on which the design, implementation, and study of 
epistemic games are based. The hypothesis proposes that any 
community of practice (e.g., biomechanical engineers) has a cul- 
ture with identifiable structuring features, including the things 
that people within the community do (skills); the understand- 
ing that people in the community share (knowledge); the way 
that members of the community see themselves (identity); the 
beliefs that members of the community hold (values); and the 
grounds that justify actions or claims as legitimate within the 
community (epistemology). For more details see Shaffer (2007) 
and Shaffer et al. (2009). 

The skills, knowledge, identities, values, and epistemologies of 
the community of practice collectively constitute that communi- 
ty's epistemic frame. The epistemic frame hypothesis argues that 
participants engage in specific forms of training and socialization 
to become members of these communities. These processes lead 
participants to internalize these epistemic frames, which they 
can in turn deploy as tools for engaging with the world from the 
point of view of a community member (Shaffer 2006). 
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A critical step in the implementation of an ECD approach is 
to conduct domain analyses in order to identify the target learn- 
ing to be assessed (the competency model and the individual 
instantiation of this model, the student model). From an epis- 
temic games perspective, the development of the competency 
model is guided by the epistemic frame hypothesis, and the 
model is constituted by those elements of the epistemic frame 
that one seeks to have players internalize. 

How these elements are organized and engaged within the 
course of the activity is based on what is known about the pro- 
fessional practica in place for training and socializing members 
of the professional group. Drawing on Schon's (1983, 1985) 
research in professional development, Shaffer et al. (2009) argue 
that epistemic frames develop and are passed on through these 
practica: "Professional practica are environments in which a 
learner takes professional action in a supervised setting and then 
reflects on the results with peers and mentors. Skills, knowl- 
edge, identity, values, and epistemology become more and more 
closely tied together as the student learns to see the world using 
the epistemic frame of the community, as happens in capstone 
courses in engineering, internship and residency for doctors, or 
almost any graduate program in the sciences" (36). 

Defining the elements of the epistemic frame to be replicated 
in the epistemic game requires that the researchers perform an 
epistemography , an in-depth study of the participant structures in 
a specific practicum environment for the purpose of identifying 
learning processes that facilitate a novice professional's develop- 
ment of a particular epistemic frame (Shaffer 2005). Participant 
structures are the forms of action and interaction, including 
reflective practices, typical of the profession. 
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In the specific case of Digital Zoo, the designers conducted an 
epistemography of an undergraduate engineering design course 
(Svarovsky and Shaffer 2006a, 2006b). Using a variety of ethno- 
graphic methods (field observations, interviews, focus groups, 
artifact archiving, and analyses) the designers followed a student 
design team as it worked to develop a biomedical device for an 
actual client. Qualitative data from student design notebooks as 
well as from regular design meetings were collected. These data 
were then analyzed using an initial coding scheme developed 
from descriptions of practice in the literature (Burghardt 1999; 
Dym and Little 2000) and the definition of an epistemic frame 
(Shaffer 2006). Through the adoption of a grounded theory 
approach (Glaser and Strauss 1967; Strauss and Corbin 1998), 
the codes were iteratively refined. 

Analysis revealed that design meetings, design notebooks, 
and meetings with clients were effective means for reflective 
activity. In particular, the design meetings and design note- 
books proved to be rich sources of epistemic statements about 
engineering that were highly correlated with references to engi- 
neering skills, knowledge, and values. To the researchers, this 
suggested that these activities played an important role in the 
initial development of students' engineering epistemic frames 
and would therefore be incorporated as important activities in 
the epistemic game. 

A storyboarding procedure (a frame board) was used to design 
and implement the systematic presentation of these activities 
(i.e., participant structures) in the game so that the students 
engaged in them in a reflexive manner. From the perspective of 
ECD, the frame board functions as the task model. Each activity 
in the frame board described the relevant participant structures, 
defining them in terms of the features of the epistemic frame 
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that are required to engage this structure at the particular point 
in the game as well as in terms of the forms of evidence neces- 
sary for claiming that these features have been developed and 
properly deployed. 

Shaffer et al. (2009) argue that the kind of professional per- 
spective that students develop from participation in epistemic 
games should not be thought of solely as a collection of the rel- 
evant skills, knowledge, values, identities, and epistemologies of 
the profession. It is also important to think of these elements as 
constituting a network of conceptual, practical, moral, personal, 
and epistemological relationships. It follows that assessing stu- 
dent learning requires not only carefully documenting instances 
of students engaging the different elements of the frame but also 
characterizing the strength of the relationships among these ele- 
ments for each student. 

One method that the Epistemic Games Research Group used 
to study these elements and their relationships was Epistemic 
Network Analysis. The approach involves a combination of 
social network analysis (Brandes and Erlebach 2005), concepts 
from frame analysis (Goffman 1974), and data visualization 
techniques. To explain the application of social network analy- 
sis in assessing learning in epistemic games, Shaffer et al. (2009) 
draw an analogy between using the technique to analyze inter- 
actions at a party and relations among the elements of an epis- 
temic frame. 

One way to examine the relationships among people meeting 
for the first time at the party would be to document at intervals 
who is interacting with whom. Assuming that a closer relation- 
ship develops among those who spend more time in the same 
conversational group, the social network that develops can be 
quantified by summing, for each pair of individuals over the 
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course of the party, the number of times they are found in the 
same conversational group. Analogously, we can think of the 
people interacting at the party as the elements from an epis- 
temic frame for one player in an epistemic game. The task then 
is to trace over time those elements of the epistemic frame that 
the player uses. 

To understand how these networks of frame elements are 
visualized, we return to the Digital Zoo epistemography. The 
documentation collected included player design notebooks, 
documents, reports, and other work products produced during 
game play. Player-mentor interactions were also recorded. For 
each player, these records were then organized into play histo- 
ries. Using working definitions of the five engineering epistemic 
frame elements (derived from the earlier qualitative coding pro- 
cess), these histories were studied to identify instances of these 
elements. 

Shaffer et al. (2009) used network graphs as tools to create 
visualizations for individual players of the epistemic frame ele- 
ments that emerged at specific moments in the individual's play 
history. These graphs showed not only which frame elements the 
player developed over time but also the nature of the relations 
among those elements. In a cumulative graph, the length of the 
line connecting the frame elements indicates the frequency with 
which each pair of elements occurred in a strip of activity: the 
closer the nodes, the more often they were linked during the 
game. Using this approach, graphs can be created for players at 
any point in the game, and these in turn can be used to perform 
developmental analyses of epistemic frames (see figure 6). 

Statistical techniques can be applied to run a variety of analy- 
ses on these epistemic network maps, providing insights into 
the development of players' epistemic frames. For example, 
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Figure 6 

Network graphs from different slices of the play history for one player 
in Digital Zoo. From Shaffer et al. (2009). 

linkages among nodes in a network can be measured at different 
time intervals to assess changes in the overall strength of asso- 
ciation of the network (weighted density). These measures can 
in turn be applied to determine changes in an epistemic frame 
over time and to associate those changes with specific elements 
of game play. 

Quest to Learn 

Levels of Analysis Individual, group, project 
Valued Outcomes Independent inquiry, collaboration 
Methods Evidence-centered design, evidence-centered evalua- 
tion, content analysis 

This example highlights assessment at all three levels for 
independent inquiry and collaboration. Although this case 
involves the assessment and evaluation of what would typically 
be considered a formal learning environment — a public school — 
both the models of assessment and evaluation applied (ECD and 
evidence-centered evaluation, or ECE) and the unusual nature 
of the curriculum around which the activities at the school are 
organized (video game design) make this project relevant also 
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for the design and implementation of assessments of informal 
learning activities. 

Quest to Learn (Q2L) is an innovative New York City pub- 
lic school whose curriculum is organized around the principles 
of game design. It is the first school of its kind, a product of 
the growing movement in education and education research to 
study and apply principles of game design to create motivating 
learning activities (see, e.g., Shaffer and Gee 2012). 

Given the school's unique curriculum, one obvious question 
that emerges is how to assess learning in an environment that 
has never been formally assessed before. Shute and Torres (2012) 
took on this challenge by adopting a two-pronged approach: to 
assess learning at the individual level and to evaluate the suc- 
cess of the Q2L project overall. They used ECE, described below, 
to evaluate and model the goals of Q2L, and ECD (previously 
discussed) to develop and implement assessments of individual 
student performance. 

Shute and Torres (2012) describe ECE as an extension of ECD. 
Both approaches are designed to guide the researcher and practi- 
tioner in specifying the structures and supporting rationales for 
the evidentiary argument of an assessment. Argument structures 
encompass, among other things, the claims (inferences) one 
wishes to make, the observables (performance data) that pro- 
vide support for those claims, the task performance situations 
that elicit the observables, and the rationales for linking them 
all together. 

As with ECD, the procedure for applying ECE involves first 
defining a set of three interrelated models for objectives, evi- 
dence, and data. The objectives model identifies the goals of the 
organization. It is a description of what is of value to the school 
and its extended community. As such, it guides the development 
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of the metrics and criteria for evaluating what works, what 
doesn't, and why. 

The first step in modeling Q2L’s goals is data collection. For 
seven months, Shute and Torres (2012) conducted semistruc- 
tured interviews, surveys, focus groups, and observations with 
and about the relevant stakeholders: teachers, students, adminis- 
trators, and curriculum designers. They also studied the school's 
design documents (Salen et al. 2010). Dominant themes were 
identified in the interview data by using content analysis. These 
themes were summarized in a document that was shared with 
all the participants. The participants were then given a survey 
to record their responses to and feedback about the document. 
After these surveys were reviewed, focus groups were organized 
as a means of further refining the themes. The Q2L objectives 
model was developed based on this set of themes (see figure 7). 

Assessment of student-level variables centered on three com- 
petencies valued by the Q2L community: systems thinking, 
teamwork, and time management. In line with an ECD approach, 



Figure 7 

Quest to Learn objectives model. From Shute and Torres (2012). 
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competency models for each variable were developed. Shute and 
Torres (2012) derived these specific models from the research 
literature, and these models were in turn used to develop corre- 
sponding assessment instruments (figure 8). 

These instruments, which were administered to Q2L students 
at six-month intervals, were just one component in a battery of 
assessments that mixed qualitative and quantitative measures. A 
qualitative observation protocol was also implemented in which 
two independent observers conducted biweekly observations in 
classrooms, the cafeteria, and the school’s after-school program. 

The focus of these observations included the following: (1) 
the thinking skills afforded by each node of the model (e.g., data 
on systems thinking, observations focused on evidence related 
to dynamic thinking, closed-loop thinking, and skills for trans- 
ferring learned models across multiple situations and settings); 
(2) the type and frequency of specialized language used within 
each domain or node; (3) the kinds of social activity evident per 
node; and (4) other relevant information, such as the learning 
tools used, the identities afforded by each site, the artifacts pro- 
duced, shared norms, physical (or virtual) space, and time allo- 
cated per activity in a given location (Shute and Torres 2012). 

The research team found that students' systems thinking 
skills improved significantly and showed signs of improvement 
in time management as well (Shute, Ventura, and Torres 2012). 

Mindful of concerns about the relationship between par- 
ticipation in the innovative activities at Q2L and achievement 
in traditional academic domains, Shute and Torres (2012) also 
planned to compare the performance on state-mandated stan- 
dardized tests between Q2L students and groups of demographi- 
cally matched students at two New York City public schools. 
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Figure 8 

Three competency models for student assessment at Quest to Learn. 
From Shute and Torres (2012). 









Highlights of the Expert Meetings 


From June 2011 through February 2012, three expert meet- 
ings with a total of 25 senior researchers were convened: at the 
Exploratorium in San Francisco, at Northwestern University in 
Evanston, Illinois, and at the University of California at San 
Diego (see appendix A). The participating researchers brought 
rich experience with project design and development, assess- 
ment and outcomes research, and external evaluations of pro- 
grammatic initiatives across a wide range of informal learning 
settings. We included scholars with experience in conceptual- 
izing and theorizing learning and assessment as well as those 
whose primary focus is conducting empirical research. 

The participants shared pertinent readings in advance of each 
meeting, and all materials were collected cumulatively on a proj- 
ect wiki. Together we viewed videos of learning in diverse settings 
and sought to connect our conceptual discussions to these con- 
crete examples. We also sought and received from the meeting 
participants suggestions on significant research projects (both 
completed and in progress), names of other leading research- 
ers to consult, and citations to relevant reports and published 
literature. This effort resulted in a bibliography (see appendix 
B), which is more extensive than the list of studies chosen for 
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review in the previous section, and it also resulted in a list of 
other relevant resources, including Web sites (see appendix C). 

To focus the scope of the report in relation to available proj- 
ect resources, we collectively decided to locate research related 
to four primary types of settings: after-school programs, com- 
munity-based organizations, museums and science centers, 
and online communities. This focus meant that we consciously 
excluded learning in the home through everyday activities or 
activities not specified by the requirements of some other edu- 
cational institution (e.g., doing and discussing mathematics 
during home remodeling). We also excluded team sports (both 
live action and "fantasy," or virtual-world mediated) and both 
online and offline hobbies communities (such as those dedi- 
cated to cooking or crafting, such as Ravelry.com). 

Certain themes recurred with varying nuance throughout the 
meetings, such as the affective dimensions of learning, learning 
at project and group levels, new modes of documentation, digi- 
tal technologies in assessment, trajectories of learning over time, 
and the diversity of kinds of valued outcomes. 

One of the key issues organizing our approach was that of 
the unit of assessment. The consensus of the group was that this 
unit is a system over time that includes individual learners, 
other participants, mediating tools, semiotic media, and local 
conditions directly relevant to and supportive of the learning 
activities. Such an analysis must extend to wider contexts that 
make the setting of learning possible institutionally, but with 
decreasing detail as relevance to the specifics of learning trajec- 
tories decreases. 

This type of approach was later applied to the question of 
metadata or backstories for video records of learning activities. A 
critical issue, the answers to which may vary by type of setting or 
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type of activity, centers on how much and what kinds of infor- 
mation are relevant to identifying valued learning and the spe- 
cific aspects of activity that support it as seen in any given video. 

In any records of learning activity over time, some valued 
learning may be more readily visible to more observers and with 
less detailed analysis or less experienced professional vision, 
whereas other instances may be less readily visible. Different 
kinds of learning may become visible when records of learning 
activities are studied over longer rather than shorter periods. 
One way to estimate the role of background information is to 
have a group view the video first without background, view it 
again with partial background, and then view it a third time 
with much more complete background. 

However, video alone is often not a sufficient documentation 
of learning activity because of the inferences that must be made 
to identify valued learning. Ideally, video should be supple- 
mented by field notes from participant observation or observant 
participation, interviews with participants, and relevant histo- 
ries of the setting and the participants. Documentation should 
cover activity in a setting for a period long enough to show the 
origins of participation, the evolution of the activity, the learn- 
ing in the activity, and the consequentiality of learning for some 
other activity. An individual episode captured on video may be 
significant as part of a longer trajectory of learning and develop- 
ment, and/or as an instance of a frequently repeated pattern of 
learning and activity in other cases in the same or similar set- 
tings. Video can be a useful tool for documenting an important 
learning event that is recognized retrospectively. 

Another key issue in the meetings was identifying the criteria 
for the value of and significance of learning. A primary criterion is 
that there is evidence of value for the participants, such as the 
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length of time they focus on a task or activity, their reluctance 
to leave or end the activity, their displays of intense or posi- 
tive affect during the activity, and any comments on the activity 
during and after explicitly elicited evaluations. 

Beyond this, there are additional criteria that may be applied, 
such as the judgment of expert educators or others (such as par- 
ents) on what is of value to the learners and/or to society and 
any evidence of consequentiality of learning for more conven- 
tional academic activities (e.g., increased interest, increased par- 
ticipation, more positive affect, more effective completion of 
tasks, the ability to teach content and skills to others, or the 
ability to solve problems collaboratively). 

In order to do assessment across settings, cumulatively or 
comparatively, we need to identify features, factors, or con- 
siderations that traverse the boundaries of the settings. Many 
common assumptions about this are flawed — identities change 
across settings, and so do modes of learning, purposes, and what 
is valued from the learning activities. Moreover, communities 
are not bound, fixed entities but are abstracted from the flow 
of practices among participants in many communities. Learning 
cannot be defined as progress toward mastery in a community, 
given this fact. Consequently, efforts to fully assess the effects 
of learning experiences must be based on longitudinal, ethno- 
graphic records, such as collections of material objects and semi- 
otic products with in-progress versions over time. 

New and promising approaches may prove to be particularly 
helpful in better understanding the diverse, widely distributed, 
and interrelated nature of learning experiences. New or under- 
utilized methods include digital storytelling as a mode of docu- 
mentation, spatial tracking, agent-based modeling, longitudinal 
assessment over periods of 5 to 10 years, the tracking of learning 
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across programs and settings, the collection and sharing of data 
about learners across programs, data mining, machine-learning 
analysis, and richly instrumented spaces. 

Many researchers are now orienting to indices of affective 
engagement, such as interest, commitment, and persistence, 
because these are assumed to be common across settings and 
therefore important for wider assessment. Others see the rela- 
tionship between learning experiences and the possibility of 
taking up subsequent opportunities as the critical element for 
assessment and look to measuring changes in the distribution 
of practices across networks as evidence of system-level learning 
(including individual learning). 

The meeting participants noted the importance of understand- 
ing learning communities in order to adequately document, assess, 
evaluate, and research them. Learning communities may differ 
from one another in their basic goals and values, strategies, and 
organization of learning as well as in the roles they make avail- 
able for members and the new niches members may create for 
themselves in the community. 

Learning communities also differ in how outcomes for indi- 
viduals, groups, and the communities as a whole are negotiated 
or established, and therefore such outcomes must be docu- 
mented, assessed, and evaluated relative to the different goals 
and values (e.g., those of individual participants, those the com- 
munity considers appropriate to various roles, those of the group 
and community itself, and those of external entities such as 
sponsoring organizations). 

Informal learning communities often differ from school- 
based learning groups in the degree to which learning outcomes 
are unpredictable — in addition to other differences such as age 
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mixing, flexible pacing, division of labor and goals according to 
role, self-guided learning, and voluntary participation. 

Within many informal learning communities, learning goals 
focus on the drivers of learning more than on the learning of 
specific content. These communities aim to improve motiva- 
tion, engagement, and enjoyment; to broaden areas of interest 
and expand zones of comfort; and to improve the skills of self- 
guided learning, sustained learning, and collaborative learning. 
Some communities and some roles may emphasize activities 
that the participants already value and enjoy, whereas others 
may draw the participants out of their comfort zones to expand 
their know-how and its range of mobilization. Some learning 
communities are also organized in such a way that groups and 
the community as a whole learn and change over time, whereas 
others are organized mainly to facilitate individual learning. 

The meeting participants engaged in an extensive discussion 
of badges as a means of recognizing achievements both in nontra- 
ditional settings and in relation to the kinds of know-how for 
which there may not be formal recognition systems. Badges 
used within communities, awarded by each community and its 
members, serve both to recognize achievement and to make the 
achievers recognizable to others as potential sources of expertise 
and assistance. Badges can also help to define ladders of partici- 
pation, which indicate for the learners the existence and nature 
of higher levels of skill. The value of a badge depends on the 
reputation of the community awarding it and the procedure by 
which it is awarded, especially if the badge is to operate and be 
recognized outside a specialized community. 

Badges are an example of the crowdsourcing of assessment. 
Within a community, this mode of evaluation and recognition 
may indicate community or group consensus rather than an 
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exercise of power and thus avoid some of the resistance-based 
invalidation of other forms of evaluative judgments. But if a 
wide-scale system of endorsement of badges or badge awarders 
(e.g., by government authority) prefers its own criteria of value 
to those of the awarding communities, it could undermine the 
authenticity of badges as endogenous evaluations and trigger 
the same reactions and invalidation seen with traditional exter- 
nal power-based evaluations (e.g., grades, standardized testing). 

Such an occurrence could lead to badge-seeking without 
engagement, badge inflation (lowering the perceived and practi- 
cal value of the badge as well as the criteria for awarding it), and 
efforts to obtain badges without durable, mobilizable learning. 

We see manipulation or degradation in recognition systems 
today with the relatively low standards for accreditation in the 
for-profit sector of higher education, the purchasing of institu- 
tions with already accredited programs, the likelihood of brib- 
ery, and so forth. It is not clear what the likely effects on an open 
badge system would be if badges were awarded by for-profit 
institutions or organizations as a source of revenue. This should 
be a major concern. 

Another issue raised was the difference between localized and 
interoperable means for assessing learning. Some methods of docu- 
mentation and procedures for assessment produce value inso- 
far as they are designed specifically in relation to the goals and 
practices of a particular community. Other methods and pro- 
cedures can be used equally well for different projects and thus 
support comparisons and generalizations. It is important to bal- 
ance these approaches in relation to the goals of local improve- 
ment and generalizable knowledge. 

For example, a coding scheme to identify a particular cate- 
gory of events in a video archive documenting a project may 
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focus on types of events that are highly specific to that project 
and its goals and that may not be in evidence or be relevant 
to any other project; or it may try to focus on types of events 
that are very likely to occur in other projects and be relevant to 
a wide range of goals. It seems desirable to try to include both 
kinds of focus in assessment. 

This example highlights the need to balance the goals of 
assessment (improvement) and the goals of research (knowledge 
relevant to future design). But the relation is not as simple as 
might be imagined. In-depth assessment of a project on its own 
terms may be of enormous value for research purposes insofar 
as it produces knowledge about how particular outcomes were 
achieved, what worked and what didn't, what was sustained 
over time and what got changed, and so on. 

This knowledge can potentially be combined with simi- 
lar knowledge from other projects to improve future designs, 
even if it was not itself set up for this purpose. Likewise, exist- 
ing research-based generalizations can be localized to serve the 
needs of assessment and improvement for particular projects 
(and in general they must be localized to be effective). 


Conclusions and Recommendations 


The principal finding and recommendation of this report is that 
the scope of valued learning outcomes for informal learning activi- 
ties should include social, emotional, and developmental outcomes as 
well as content knowledge and should include learning by groups and 
whole projects as well as by individuals. We note that many of the 
valued learning outcomes that are reported were not predictable 
or aimed for at the start of the projects. 

Effective documentation and assessment of informal learn- 
ing activities should observe several general principles as much 
as possible. Such assessment should take into account the social 
and cultural context of the communities and institutions that 
support and constrain the activities being assessed. The assess- 
ment of informal learning activities must be specific to the goals 
of each project and activity and take account of the history of 
the project, its supporting organizations, and the surrounding 
community. It should aim to provide insight into how and why 
various features of the projects and activities have or have not 
supported valued learning outcomes and learning in progress for 
individuals, groups, and whole projects or organizations. 

Effective documentation and assessment should also take into 
account community acceptance and the prospects for long-term 
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sustainability, aiming to provide a basis for estimating the likeli- 
hood that a project is sustainable under various expectable con- 
ditions and the likelihood that its successful practices can be 
adapted to other specified contexts. Key indices of sustainability 
include the forms and extent of community cosponsorship and 
financial and in-kind contributions. 

Overall, we believe that the unit of analysis for assessment 
should be a system of activities and practices and take place 
on multiple time scales — from hours to months or years. These 
systems may be a cluster of related activities across a range of 
time scales: a single instance; a sequence of activities that build 
on one another's content, themes, and skills; the duration of a 
working group; the entire time of an individual's participation in 
the project; or the full time span of the project. This means that 
the assessment should be built on the recognition that groups 
and projects as well as individuals learn to do better and that it 
should aim to generate information that will be of practical use 
to individual learners, groups, projects, and organizations. 

To attend to multiple time scales, assessment for informal 
learning activities should adopt a longitudinal design, following 
changes and gains from the inception of a project and document- 
ing the processes by which desirable outcomes occur in addi- 
tion to the outcomes themselves. Such longitudinal approaches 
should include consideration of the processes of improvement 
on the part of groups and teams, whole projects, and partici- 
pating organizations over the range of time scales, from what 
is learned by individuals in an hour to what may be learned by 
organizations over several years. 

Professional quality assessment should provide information 
and interpretations that have practical value for individuals in 
gauging their changing strengths and weaknesses, for groups in 
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gauging and improving their effectiveness, and for projects and 
organizations in determining whether their goals are being met. 
The results should be represented in ways that allow for changes 
that better support the learning and development of partici- 
pants, of groups, and of the project or organization as a whole. 

Assessment information is useful not only in identifying out- 
comes but also in planning for future improvements for the 
design and development of learning environments. Assessments 
that are made only at the end of a project are not likely to be 
sensitive to learning about how to do things better, nor can they 
identify the processes by which improvements came to be made. 
Assessments based only on the initial goals of a project may not 
capture other valued learning outcomes that emerged unpredict- 
ably over time. Documentation and assessment that sample out- 
comes only on short time scales (e.g., by standardized testing) 
cannot assess practices that inherently take much longer to be 
enacted (e.g., designing a product or refining an experiment). 

Assessment must also be organized so that it can recognize 
unanticipated valuable outcomes and processes in progress. We pro- 
pose a synthesis of assessment approaches that encompasses 
many general types of valued outcomes, to be assessed at each 
of three organizational levels: project, group, and individual 
(see Table 1). We find that outcomes at each level are frequently 
influenced by those at other levels, such as when groups learn 
how to better support the learning of individual members and 
when projects learn how to better enable participants to identify 
and achieve their own learning goals. 

Valued outcomes include more than just acquired knowledge. 
The definition of knowledge for assessing informal learning 
should be broad enough to include know-how and know-who 
as well as know-that. The assessment should examine evidence 
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that knowledge is being used (knowing how to take the next step 
in an activity) and that this use persists, grows, and cumulates 
over relatively long periods. Relevant knowledge includes such 
socially oriented capabilities as assisting others to achieve valued 
learning outcomes, completing tasks cooperatively, and know- 
ing how to build relationships and negotiate social networks. 

In addition to knowing how to collaborate with others, 
important capabilities in most settings include knowing who 
can contribute to meeting a need (know- who). Important out- 
comes also include the capacity to mobilize learning across a 
widening range of tasks, domains, and settings and an improve- 
ment in one's ability to successfully guide one's own learning. 

Other forms of socioemotional development include increased 
emotional maturity, the productive use of affective sensibilities, 
and comfort with and sense of agency in a domain. Informal 
learning environments typically seek to support participants' 
self-respect, responsibility, initiative, and sense of agency. These 
might be assessed in conjunction with forms of knowledge and 
skills particular to epistemic frames or content domains, such 
as the capacity to generate high-quality products or to evaluate 
the credibility of claims and information sources. In learning 
environments well attuned to the social and cultural context, it 
would be valuable to document learners' persistence and resil- 
ience in the face of specific obstacles, setbacks, and frustrations 
and their continued development and application of learning 
over long stretches of time. 

Documentation activities should support the assessment of 
both anticipated and unanticipated outcomes as well as record- 
ing the processes over time by which the outcomes are produced. 
In the review of the literature provided in this report, we high- 
light the variety of methods of documentation and assessment 
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that have proven valuable or are considered promising and wor- 
thy of further support and investigation. 

Among these are broad approaches, such as Evidence Cen- 
tered Design (ECD) of assessment and methods derived from it, 
the use of video documentation and both close analysis and cat- 
egory-based data mining methods of video archives, embedded 
and unobtrusive integrated assessment in learning games and 
simulations, longitudinal collections of learner and group prod- 
ucts, ethnographic observation and field notes, and interviews 
with participants and organizational stakeholders. 

In addition to the accounts of particular projects reviewed 
here, we have also included a comprehensive bibliography that 
focuses on projects and discussions of assessment of informal 
learning activities that have been influential or deserve to be, 
many of which were recommended to us by members of our 
expert panels and from which we made our selection of the stud- 
ies to be reviewed . 1 

Promising Directions and Recommendations 

In the course of the review of the literature and the meetings of 
our expert panels, we identified promising new directions for 
informal learning activities, projects, and methods of documen- 
tation and assessment that are deserving of support and further 
investigation into their effectiveness and usefulness. 

These efforts include projects developed and implemented 
by partnerships between sponsoring organizations (such as uni- 
versities and informal science education institutions) and local, 
community-based organizations whose leaders and members 
can represent the perspectives and perceived needs of the com- 
munity in which the project will operate. These projects often 
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Box 2 

A Comment on Standardized Testing 


In the projects we have reviewed, standardized measures and test 
scores play a very small role in assessment, and in the discussions 
in our expert panel meetings, there was little or no sentiment 
expressed favoring the relevance or greater use of such measures. 

Our view is that the validity of standardized testing measures 
has not been established for informal learning activities, in large 
part because these activities do not have fixed, predictable cur- 
ricular outcomes and because the valued outcomes they do have 
are either unpredictable or do not have existing valid standard 
measures that can be used in the informal learning context 
without being disruptive of other goals and commitments (e.g., 
enjoyment, creative production, maintenance of a free-choice 
community). Thus their use, though not excluded and sometimes 
of value, should not be considered a norm or an ideal of good 
assessment for informal learning. In fact, when they are used, the 
outcome may say as much about whether the test fits the activ- 
ity (i.e., its content and context validity for this particular use) as 
whether the activity is producing higher scores on what the test 
tries to measure. 

Policymakers and funders do need assessments that provide a 
basis for comparison across projects and approaches. The devel- 
opment of such assessments for informal learning activities will 
require research that can compare and align project-specific 
assessments across multiple sites and projects over time (see 
below). It may also be useful to identify the common character- 
istics of successful informal learning activities across projects and 
use these as benchmarks when considering new proposals, while 
still encouraging innovation. 
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expand the number and kinds of mentors (such as undergradu- 
ates, older students, and noneducation professionals), rethink- 
ing the roles of those who can assist in activities and redesigning 
their training and experience. 

We also see promise in the development of comprehen- 
sive documentation of activities and processes in richly instru- 
mented spaces designed for assessment purposes. One mode for 
this is computer-assisted learning games that automatically and 
unobtrusively document learning progress. These can have a 
wide range of application if they can be accessed through mobile 
computing devices (phones and tablets), even if they run on a 
remote server. 

Such approaches lend themselves to data mining and 
machine-learning analysis of logs of interaction with computer 
programs, video archives, field note archives, and databases of 
participant products to identify patterns of successful and unsuc- 
cessful practices on the part of participants and project organiz- 
ers. Ethnographic research in online virtual worlds, where new 
kinds of social learning are taking place, can lay the foundation 
for developing comprehensive documentation and assessment 
in these new environments. Data mining approaches must be 
accompanied by the development of tools to analyze the wealth 
of data collected in new and useful ways. 

Other new approaches show promise as well — for example, 
the use of agent-based modeling to interrogate and attempt to 
simulate the key processes and practices of participants that 
contribute to valued outcomes. The crowdsourcing of assess- 
ment (e.g., the awarding of badges that recognize achievement 
by peers or senior peers in a project community) can enlist the 
knowledge and judgment of those closest to an activity. Care 
should be taken, however, that peer judgments are not super- 
seded by external, standardized criteria. 
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The adaptation of the ECD model to planning more com- 
prehensive documentation and assessment of projects provides 
another avenue for advancing the field. In particular, the the- 
oretical requirements in ECD for developing assessment argu- 
ments help make explicit the types of considerations pertinent 
to informal learning settings as they are discussed in this report. 

In addition to our general conclusions and findings on prom- 
ising practices, we have developed more specific recommenda- 
tions for the research, practice, and funding communities: 

• We suggest that funding for informal learning projects should 
include set-aside funds for professional evaluation and assess- 
ment — to begin as early in the work of the project as possible, 
to document learning across the range of outcomes and levels 
identified in this report, to develop and apply project-specific and 
activity-specific measures, to modify these as new elements in the 
project emerge, and to iteratively improve the local assessment 
model in close interaction with project participants in all roles. 

• We recommend follow-up assessments of the valued conse- 
quences of participating in projects one year or more after the 
end of participation. We believe that longitudinal and ongoing 
assessment of projects with multiyear durations is necessary, 
with a focus on change, improvement, and sustainability. 

• We see particular value in the same assessment team having 
the opportunity to work with more than one project in order to 
help develop indices and measures with validity for more than 
one project, or to document why this strategy may have limited 
usefulness. 

• We further support the development of data mining and 
learning analytics tools that can be applied to a range of data 
types, including video and field note documentation archives 
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and participant work portfolios, and that can produce results 
that have practical, useful interpretations. 

• Research on the nature and distinctive features of infor- 
mal learning communities and how they may differ from 
school-based learning groups is important for improving our 
understanding of social learning in informal activities and for 
estimating the transferability of successful practices from infor- 
mal to formal settings. It also would be valuable to have more 
research that examines the relationship between play and learn- 
ing, and learning trajectories in activities in which playfulness 
and enjoyment are the dominant mood and motive, rather than 
more exclusively serious approaches to learning for its own sake. 

We wish to thank the MacArthur Foundation for sponsoring 
this effort and the many generous colleagues who have contrib- 
uted to deepening our understanding. The effective documenta- 
tion and assessment of informal learning activities has much to 
teach us about how and why a wide range of valued outcomes 
result from such activities. It may have much to contribute in 
the future to rethinking assessment in formal education as well. 
We hope that this report provides a basis for thoughtful discus- 
sion, a useful account of the current state of the art, and a chal- 
lenge to all of us to do better. 
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California Academy of Sciences, http://www.calacademy.org 

Center for Assessment, http://www.nciea.org 

Center for Informal Learning and Schools. http://cils.explora 

torium.edu 

Coalition for Science after School. http://www.afterschoolscience. 
org 

Digital Media Learning Central, http://dmlcentral.net 
Encyclopedia of Informal Education, http://www.infed.org/ 
index.htm 

Exploratorium. http://www.exploratorium.edu 
Futurelab. http://www.futurelab.org.uk 


142 


Appendix C 


Harvard Family Research Project, http://www.hfrp.org 
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•org 

Relating Research to Practice: Current Research in Science Edu- 
cation. http://relatingresearchtopractice.org 
Remediating Assessment. http://remediatingassessment.blogspot 
.com 

STELAR (STEM Learning and Research Center), http://stelar.edc 
.org 

Transformative Learning Technologies Laboratory. http://tltl 
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U.S. Department of Education, http://www.ed.gov 
U.S. Department of Education, Office of Educational Technol- 
ogy. http://tech.ed.gov 

U.S. Department of Education — Reports, http://www2.ed.gov/ 
about/ pubs/ publications-reports . html 

William T. Grant Foundation. http://www.wtgrantfoundation 
•org 

Examples of Out-of-School Learning Projects 

Computer Clubhouse Network. http://www.computerclubhouse 
•org 

Digital Youth Network, http://digitalyouthnetwork.org 

FabLab@School. http://tltl.stanford.edu/projects/fablabschool 

Games for Learning Institute, http://g41i.org 

Healthy Youth Peer Education Program (HYPE). http://www 

.hypeprogram.org 
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Partnerships for Informal Science Education in the Communityl 
CU-Boulder. 

http : / / jila-pf c. colorado.edu/ pisec/home 
Quest to Learn, http://www.q21.org 

Stanford Makers' Club, http://stanfordmakersclub.ning.com 
University-Community Links, http://uclinks.berkeley.edu 
You Media, http://youmediachicago.org 

YouSTEM: Interest-Driven STEM Exploration Space. http://osep 
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Youth Media International, Youth Radio. http://www.youth 
radio.org 


Notes 


Review of the Literature 

1. Scratch 2.0 was released in May 2013. 

2. The platform for the online social network is Remix World (http:// 
remixlearning.com). Developed by Nichole Pinkard and her colleagues, 
the platform was designed specifically for use by educational institu- 
tions — schools, museums, after-school centers — and is customizable to 
fit the particular needs of each institution. 

3. In relation to a broader analysis of the social organization of learning 
opportunities, Barron et al. (2014) discuss these and other mixed- 
method approaches for studying the imaginative production, expertise, 
and relationships promoted in the face-to-face and online YouMedia 
environments. 

Conclusions and Recommendations 

1. Our bibliography does not focus on the evolving theoretical litera- 
ture on validity in assessment, although we recognize its relevance and 
the convergence of some strands of this work with our own 
conclusions. 
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